KR20230070976A - Apparatus for generating image dataset for training and evaluation of image stitching and method thereof - Google Patents

Abstract

Disclosed are an image dataset generation device for image stitching learning and evaluation and a method therefor. According to the present disclosure, the image dataset generation device sets a parameter for a reference camera and a target camera in a three-dimensional dataset, extracts a reference image and a target image based on the set parameter, and further generates an occlusion mask, an overlap mask, and a warp vector. Therefore, the present invention is capable of promoting a development of an algorithm related to image stitching.

Description

Image dataset generation apparatus and method for image stitching learning and evaluation

The present disclosure relates to an image dataset generation technology for learning and evaluating a model performing image stitching, and specifically, it can be used for learning and evaluating an image stitching model in an existing dataset containing 3D information of an object. It is about a technique to create a new dataset that exists.

Image stitching is a method of generating a single panoramic image or a high-resolution image by combining images having overlapping regions captured at various viewpoints. Images synthesized through image stitching can be used in various fields such as robot navigation, high-resolution satellite images, and high-resolution medical images.

Accordingly, attempts to apply deep learning technology to image stitching are actively being made, and a dataset to be used for training and evaluation of a deep learning model that naturally performs image stitching is required.

However, as the field of image stitching is a field where deep learning is newly applied, there are almost no datasets for the main purpose of image stitching, and the datasets used for training and evaluation of existing deep learning models are There is a problem that is not suitable for application to the image stitching model because the purpose of is different.

The present disclosure is proposed to solve the above-described problems, and extracts images from a 3D dataset and generates a correct answer (GT; Ground Truth) corresponding to each image. A dataset for learning and evaluating an image stitching model It is an object of the present invention to provide a method for generating and a device using the same.

More specifically, the present disclosure extracts a reference image and a target image by setting parameters of a reference camera and a target camera in a 3D dataset, and sets an occlusion mask and an overlap mask corresponding to the reference image and the target image. An object of the present invention is to provide a method for generating a mask and a warp vector and a device using the same.

The technical problem to be achieved by the present disclosure is not limited to the technical problems as described above, and other technical problems may be inferred from the following embodiments.

A method for generating an image dataset according to an embodiment disclosed herein includes setting one or more parameters for a reference camera and a target camera in a 3D dataset; extracting a reference image and a target image from the 3D dataset based on at least some of the set parameters; generating an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters; and generating an overlap mask between the reference image and the target image and a warp vector from the target image to the reference image, based on at least some of the set parameters.

According to an embodiment, the one or more parameters may include position coordinates of the reference camera, position coordinates of the target camera, Euler angle of the reference camera, Euler angle of the target camera, and focal length of the reference camera. , a focal length of the target camera, a resolution of the reference camera, a resolution of the target camera, a projection matrix of the reference camera, and a projection matrix of the target camera.

According to an embodiment, the extracting may include extracting the reference image by adjusting position coordinates and Euler angles of the reference camera, and extracting the target image by adjusting position coordinates and Euler angles of the target camera. can be characterized.

According to an embodiment, the extracting may include scaling and summing unit vectors randomly extracted within a preset range to the position coordinates of the reference camera and set values of Euler angles, thereby adding the position coordinates of the reference camera. and Euler angles, and adjusting the position coordinates of the target camera and Euler angles by scaling and adding randomly extracted unit vectors within a preset range to each of the set values of the target camera and Euler angles. can be done with

According to an embodiment, the generating of the occlusion mask may include projecting a first ray at the location coordinates of the reference camera and generating a second ray at the location coordinates of the target camera based on a ray casting algorithm. It may be characterized in that the occlusion mask is generated by projecting.

According to an embodiment, the generating of the occlusion mask may include projecting the first light ray in a direction indicated by each pixel of the reference image at position coordinates of the reference camera; calculating the coordinates of an object that the first ray first encounters in a traveling path as first coordinates; projecting the second light ray toward the first coordinates from the location coordinates of the target camera; calculating the coordinates of an object that the second ray encounters first in a traveling path as second coordinates; and generating the occlusion mask based on whether the first coordinates match the second coordinates.

According to an embodiment, the generating of the occlusion mask may include allocating 1 to a pixel of the occlusion mask corresponding to a pixel of the reference image when the first coordinate and the second coordinate match, and When the first coordinate and the second coordinate do not match, 0 may be assigned to a pixel of the occlusion mask corresponding to a pixel of the reference image.

According to an embodiment, the generating of the overlapping mask and the warp vector may include generating the overlapping mask and the warp vector based on a projection matrix of the reference camera and a projection matrix of the target camera. there is.

According to an embodiment, the generating of the overlap mask and the warp vector may include calculating an inverse matrix of a projection matrix of the reference camera; calculating three-dimensional coordinates for each pixel of the reference image by matrix-multiplying the coordinates of each pixel of the reference image with the inverse matrix; Calculating projection coordinates from the target camera to each pixel by matrix multiplication of the projection matrix of the target camera by the three-dimensional coordinates of each pixel of the reference image; generating the overlapping mask based on whether each of the projection coordinates is included in the target image; and generating the warp vector based on the coordinates of each pixel of the reference image and the projection coordinates.

According to an embodiment, the generating of the overlapping mask may include allocating 1 to a pixel of the overlapping mask corresponding to a pixel of the reference image when the projection coordinates are included in the target image, and the projection coordinates are When not included in the target image, 0 may be assigned to pixels of the overlapping mask corresponding to pixels of the reference image.

According to an embodiment, the generating of the warp vector may include generating the warp vector by subtracting each of the projected coordinates from the coordinates of each pixel of the reference image.

According to an embodiment, the occlusion mask, the overlap mask, and the warp vector train and evaluate a deep learning model that performs image stitching on the reference image and the target image. It may be characterized in that it is used as a correct answer (GT; Ground Truth) in the process.

An apparatus for generating an image dataset according to an exemplary embodiment includes one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs, in a 3D data set, are configured to operate on a reference camera and a target camera. instructions for setting one or more parameters for; a command for extracting a reference image and a target image from the 3D dataset based on at least some of the set parameters; an instruction for generating an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters; and instructions for generating an overlap mask between the reference image and the target image and a warp vector from the target image to the reference image, based on at least some of the set parameters. can

A computer program for generating an image dataset according to an embodiment disclosed herein includes, in combination with hardware, setting one or more parameters for a reference camera and a target camera in a 3D dataset; extracting a reference image and a target image from the 3D dataset based on at least some of the set parameters; generating an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters; and generating an overlap mask between the reference image and the target image and a warp vector from the target image to the reference image based on at least some of the set parameters. It can be stored in a computer readable recording medium.

A non-transitory computer readable storage medium according to an embodiment disclosed herein includes a medium configured to store computer readable instructions, and when the computer readable instructions are executed by a processor, the processor: 3D data set setting one or more parameters for a reference camera and a target camera in; extracting a reference image and a target image from the 3D dataset based on at least some of the set parameters; generating an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters; and generating an overlap mask between the reference image and the target image and a warp vector from the target image to the reference image, based on at least some of the set parameters. Data set creation method can be performed.

Details of other embodiments are included in the detailed description and drawings.

According to the present disclosure, image-specific features (eg, occlusion mask, overlapping mask, warp vector) required for learning and evaluation of an image stitching model are generated using only a 3D dataset as an input to construct a dataset, thereby image stitching The cost and time required to build a dataset suitable for the field can be greatly reduced, and furthermore, the development of algorithms related to image stitching can be promoted.

Effects of the invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a schematic configuration diagram illustrating a system for generating an image dataset according to an exemplary embodiment.
2 is a flowchart illustrating a method of generating an image dataset according to an exemplary embodiment.
3 is a flowchart for describing in detail a step of generating an occlusion mask according to an exemplary embodiment.
4 is a flowchart for explaining in detail a step of generating an overlap mask and a warp vector according to an embodiment.
5 is a block diagram illustrating an apparatus for generating an image dataset according to an exemplary embodiment.

The terms used in the embodiments have been selected as general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. In addition, in a specific case, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the corresponding description. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

When it is said that a certain part "includes" a certain component throughout the specification, it means that it may further include other components without excluding other components unless otherwise stated. In addition, terms such as "...unit" and "...module" described in the specification mean a unit that processes at least one function or operation, which is implemented as hardware or software or a combination of hardware and software. It may be, and unlike the illustrated example, it may not be clearly distinguished in specific operations.

The expression of "at least one of a, b, and c" described throughout the specification means 'a alone', 'b alone', 'c alone', 'a and b', 'a and c', 'b and c' ', or 'all a, b, and c'.

A “terminal” referred to below may be implemented as a computer or portable terminal capable of accessing a server or other terminals through a network. Here, the computer includes, for example, a laptop, desktop, laptop, etc. equipped with a web browser, and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution), etc. It may include a handheld-based wireless communication device.

In the following description, terms such as "transmission", "communication", "transmission", "reception" and other similar meanings of signals or information refer not only to direct transmission of signals or information from one component to another, but also to It also includes passing through other components.

In particular, "transmitting" or "transmitting" a signal or information as a component indicates the final destination of the signal or information, and does not mean a direct destination. The same is true for "reception" of signals or information. Also, in this specification, two or more data or information being “related” means that when one data (or information) is obtained, at least a portion of other data (or information) can be obtained based thereon.

Also, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another.

For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

In describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to more clearly convey the gist of the present invention without obscuring it by omitting unnecessary description.

For the same reason, in the accompanying drawings, some components are exaggerated, omitted, or schematically illustrated. Also, the size of each component does not entirely reflect the actual size. In each figure, the same reference number is assigned to the same or corresponding component.

Advantages and features of the present invention, and methods for achieving them, will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

It will be appreciated that each block of process flow diagrams and combinations of flow diagrams may be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, so that the instructions executed by the processor of the computer or other programmable data processing equipment are described in the flowchart block(s). It creates means to perform functions. These computer program instructions may also be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular way, such that the computer usable or computer readable memory The instructions stored in are also capable of producing an article of manufacture containing instruction means that perform the functions described in the flowchart block(s). The computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to generate computer or other programmable data processing equipment. Instructions for performing processing equipment may also provide steps for performing the functions described in the flowchart block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative implementations it is possible for the functions mentioned in the blocks to occur out of order. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in reverse order depending on their function.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

1 is a schematic configuration diagram illustrating a system for generating an image dataset according to an exemplary embodiment.

As shown, the image dataset generating system 100 according to an embodiment includes an image dataset generating device 110 . In addition, it can be understood by those skilled in the art that other general-purpose components may be further included in addition to the components shown in FIG. 1 .

The image dataset generating device 110 takes a 3D dataset as an input and generates a dataset consisting of an image and a corresponding ground truth (GT) for learning and evaluation of a model that performs image stitching. means

According to an embodiment, the image dataset generating device 110 may acquire a 3D dataset from the outside through a wired or wireless network. More specifically, the 3D dataset may be a dataset of a 3D space captured by a photographing device represented by a camera, a dataset pre-stored in a cloud server, or a dataset pre-stored in a local database (DB). . However, this is just an example, and the 3D spatial imaging means for generating the 3D data set or the storage location of the 3D data set may be set in various ways according to embodiments.

According to an embodiment, after generating an image dataset, the image dataset generating device 110 stores the created image dataset in a separate storage medium or in a storage space within the image dataset generating device 110. Alternatively, it can be stored again in a cloud server or local DB.

Meanwhile, according to an embodiment, a wired or wireless network used to transmit an input to the image dataset generating device 110 and an output from the image dataset generating device 110 is a local area network (LAN). ), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and a mutual combination thereof, each network configuration shown in FIG. It is a comprehensive data communication network that enables subjects to communicate smoothly with each other, and may include wired Internet, wireless Internet, and mobile wireless communication network. Wireless communication includes, for example, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy (Bluetooth low energy), Zigbee, WFD (Wi-Fi Direct), UWB (ultra wideband), infrared communication (IrDA, Infrared Data Association) ), Near Field Communication (NFC), etc., but is not limited thereto.

In relation to the above, it will be described in more detail through the following drawings.

2 is a flowchart illustrating a method of generating an image dataset according to an exemplary embodiment.

The method shown in FIG. 2 may be performed, for example, by the above-described image dataset generating apparatus 110.

In step S210, the image dataset generating apparatus 110 may set one or more parameters for a reference camera and a target camera in the 3D dataset.

According to an embodiment, the image dataset generating apparatus 110 may set at least some of the following parameters for a reference camera.

(1) Positional coordinates of the reference camera

(2) Euler angle of reference camera

(3) Focal length of reference camera

(4) Resolution of reference camera

(5) Projection matrix of reference camera

According to an embodiment, the image dataset generating apparatus 110 may set at least some of the following parameters for a target camera.

(1) Positional coordinates of the target camera

(2) Euler angle of target camera

(3) Focal length of target camera

(4) Resolution of target camera

(5) Projection matrix of target camera

For example, the image dataset generation apparatus 110 sets the location coordinates of the reference camera to (-30, 0, 0) and the location coordinates of the target camera to (30, 0, 0), and sets the Euler of each camera to You can set the angle to (0, 0, 0).

Meanwhile, more specifically, in setting the projection matrix of each camera, the image dataset generating apparatus 110 uses an intrinsic matrix derived from parameters corresponding to the intrinsic characteristics of each camera and a camera A projection matrix of each camera may be set by multiplying an extrinsic matrix derived from an extrinsic parameter corresponding to an extrinsic characteristic. In this regard, illustratively, the internal matrix of the camera may be derived based on the focal length and resolution of the camera, and the external matrix of the camera may be derived based on the position coordinates and Euler angles of the camera. The projection matrix thus set can be multiplied with the 3-dimensional coordinates to calculate 2-dimensional coordinates obtained by projecting the 3-dimensional coordinates onto a plane.

In step S220, the image dataset generating device 110 may extract a reference image and a target image from the 3D dataset based on at least some of the set parameters. Since correct answers (occlusion mask, superposition mask, warp vector) corresponding to the reference image and the target image are generated through steps S230 and S240 to be described later, the image dataset generating apparatus 110 performs several reference images and target images through step S220. By extracting , it is possible to construct a large dataset for learning and evaluating image stitching models.

In the present disclosure, a 'reference image' refers to an image extracted based on parameters set for a reference camera, and a 'target image' refers to an image extracted based on parameters set for a target camera and after undergoing a predetermined transformation. This means an image to be stitched to the reference image.

According to an embodiment, the image dataset generating apparatus 110 may extract a reference image by adjusting position coordinates and Euler angles of a reference camera among set parameters, and may extract a target image by adjusting position coordinates and Euler angles of a target camera. can be extracted.

In this regard, in detail, the image dataset generation apparatus 110 scales and sums randomly extracted unit vectors within a preset range to each of the position coordinates of the reference camera and set values of Euler angles, thereby scaling the position of the reference camera. Coordinates and Euler angles can be adjusted. In addition, the image dataset generation device 110 may adjust the position coordinates of the target camera and Euler angles by scaling and summing randomly extracted unit vectors within a preset range to each set value of the position coordinates and Euler angles of the target camera. there is.

For example, the image dataset generating device 110 may adjust the location coordinates of the camera based on Equation 1 below.

[Equation 1]

는 k번째로 조절된 카메라의 위치 좌표,

는 단계 S210에서 설정된 카메라의 위치 좌표,

는

의 크기를 조절하기 위한 스케일 팩터(scale factor),

는 0 이상 N-1 이하의 정수(이때, N은 추출할 이미지의 개수),

는 카메라의 위치 좌표 조절을 위해 k번째로 무작위로 추출된 3차원 벡터를 나타낸다.At this time,

is the coordinates of the position of the k-th adjusted camera,

is the location coordinate of the camera set in step S210,

Is

A scale factor for adjusting the size of

is an integer from 0 to N-1 (where N is the number of images to be extracted),

represents a k-th randomly extracted 3D vector for adjusting the position coordinates of the camera.

를 추출할 수 있다.According to an embodiment, the image dataset generation apparatus 110 randomly extracts each vector component in the range of -1 or more and 1 or less in relation to Equation 1 described above.

can be extracted.

를 추출할 수 있다.According to another embodiment, in relation to Equation 1 above, the image dataset generation apparatus 110 takes the right direction of the camera as the x-axis, the upward direction of the camera as the y-axis, and the direction the rear of the camera faces. After setting to the z-axis, based on Equation 2 below

can be extracted.

[Equation 2]

는 x축에 대한 단위 벡터,

는 y축에 대한 단위 벡터,

는 0 이상 2π 이하에서 무작위로 선택한 값을 나타낸다.At this time,

is the unit vector about the x-axis,

is the unit vector about the y-axis,

represents a randomly selected value from 0 to 2π.

Meanwhile, for example, the image dataset generating device 110 may adjust the Euler angle of the camera based on Equation 3 below.

[Equation 3]

,

,

은 오일러 각의 각 성분을 나타낸다.At this time,

,

,

represents each component of Euler angles.

의 각 성분을 -1 이상 1 이하의 범위에서 무작위로 추출하고,

로

의 크기를 조절함으로써 카메라의 오일러 각을 조절할 수 있다.According to an embodiment, the image dataset generating apparatus 110 relates to Equation 3 described above,

Each component of is randomly extracted in the range of -1 or more and 1 or less,

as

You can adjust the Euler angle of the camera by adjusting the size of .

In operation S230, the image dataset generating apparatus 110 may generate an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters.

According to an embodiment, the image dataset generation apparatus 110 projects a first ray at the position coordinates of a reference camera and projects a second ray at the position coordinates of a target camera based on a ray casting algorithm, You can create an occlusion mask.

According to an embodiment, the occlusion mask generated by the image dataset generating device 110 is used for training and evaluation of a deep learning model that performs image stitching on a reference image and a target image. It can be used as the correct answer (GT; Ground Truth) in the process.

In this regard, it will be examined in more detail with reference to FIG. 3 below.

3 is a flowchart for describing in detail a step of generating an occlusion mask according to an exemplary embodiment.

The method shown in FIG. 3 may be performed, for example, by the above-described image dataset generating apparatus 110.

In operation S310, the image dataset generating apparatus 110 may project a first ray in a direction indicated by each pixel of the reference image from the location coordinates of the reference camera.

Specifically, since the reference image constitutes one plane in the 3D space, coordinates are assigned to each pixel of the reference image in the 3D space. That is, the image dataset generating apparatus 110 may project a first ray toward coordinates assigned to each pixel of the reference image from the position coordinates of the reference camera.

In step S320, the image dataset generating apparatus 110 may calculate the coordinates of an object that the first ray first encounters in the traveling path as the first coordinates.

In operation S330, the image dataset generating apparatus 110 may project a second light ray toward the first coordinates from the location coordinates of the target camera.

In step S340, the image dataset generating apparatus 110 may calculate the coordinates of an object that the second ray first encounters in the traveling path as second coordinates.

In step S350, the image dataset generating apparatus 110 may generate an occlusion mask based on whether the first coordinates and the second coordinates match.

According to an embodiment, when the first coordinates and the second coordinates match, the image dataset generation apparatus 110 assigns 1 to a pixel of the occlusion mask corresponding to a pixel of the reference image, and assigns 1 to the first coordinate and the second coordinate. If the coordinates do not match, the occlusion mask may be created by assigning 0 to pixels of the occlusion mask corresponding to pixels of the reference image.

Referring back to FIG. 2 , in step S240, the image dataset generating apparatus 110 determines an overlap mask between a reference image and a target image and a target image to a reference image based on at least some of the set parameters. You can create warp vectors.

In the present disclosure, a 'warp vector' refers to a vector representing a degree of movement of each pixel on an image when the image is transformed from one coordinate system to another. In addition, an 'overlapping mask' means a matrix indicating whether pixels of an image to be converted are present in the reference image.

According to an embodiment, the image dataset generating apparatus 110 may generate an overlap mask and a warp vector based on a projection matrix of a reference camera and a projection matrix of a target camera.

According to an embodiment, the overlapping mask and warp vector generated by the image dataset generating device 110 may be used as correct answers in a process of learning and evaluating a deep learning model that performs image stitching on a reference image and a target image. there is.

In this regard, it will be examined in more detail with reference to FIG. 4 below.

4 is a flowchart for explaining in detail a step of generating an overlap mask and a warp vector according to an embodiment.

The method shown in FIG. 4 may be performed, for example, by the above-described image dataset generating apparatus 110.

In step S410, the image dataset generating apparatus 110 may calculate an inverse matrix of the projection matrix of the reference camera.

According to an embodiment, the image dataset generating apparatus 110 may calculate a Moore-Penrose pseudoinverse matrix for the projection matrix of the reference camera. Through this, the image dataset generating apparatus 110 may calculate an inverse matrix for a projection matrix having an arbitrary size even when the projection matrix of the reference camera is not a square matrix.

In operation S420, the image dataset generating apparatus 110 may calculate 3D coordinates for each pixel of the reference image by matrix-multiplying the coordinates of each pixel of the reference image with an inverse matrix of the projection matrix of the reference camera.

In operation S430 , the image dataset generating apparatus 110 may calculate projection coordinates from the target camera to each pixel by multiplying the projection matrix of the target camera by the 3D coordinates of each pixel of the reference image.

In operation S440, the image dataset generating apparatus 110 may generate an overlapping mask based on whether each of the calculated projection coordinates is included in the target image.

According to an embodiment, when the projected coordinates are included in the target image, the image dataset generating apparatus 110 assigns 1 to pixels of an overlapping mask corresponding to pixels of the reference image, and the projected coordinates are not included in the target image. If not, an overlapping mask can be created by assigning 0 to pixels of the overlapping mask corresponding to pixels of the reference image.

In operation S450, the image dataset generating apparatus 110 may generate a warp vector based on the coordinates and projection coordinates of each pixel of the reference image.

According to an embodiment, the image dataset generating apparatus 110 may generate a warp vector by subtracting each projected coordinate from the coordinates of each pixel of the reference image. In other words, the image dataset generating apparatus 110 may generate a warp vector from the target image to the reference image for each pixel of the reference image.

2 to 4, the method is divided into a plurality of steps, but at least some steps are performed in reverse order, combined with other steps, performed together, omitted, or divided into detailed steps. Or, one or more steps not shown may be added and performed. In particular, steps S440 and S450 of FIG. 4 may be performed simultaneously or in reverse order according to an embodiment.

5 is a block diagram illustrating an apparatus for generating an image dataset according to an exemplary embodiment.

The image dataset generating device 110 may include a transceiver 111 , a processor 113 , and a memory 115 , according to an embodiment. The image dataset generating device 110 may be connected to other external devices through the transceiver 111 and exchange data.

The processor 113 may perform at least one method described above through FIGS. 1 to 4 . The memory 115 may store information for performing at least one method described above through FIGS. 1 to 4 . Memory 115 may be volatile memory or non-volatile memory.

The processor 113 may control the image dataset generating device 110 to execute a program and provide information. Program codes executed by the processor 113 may be stored in the memory 115 .

The processor 113 is connected to the transceiver 111 and the memory 115, sets one or more parameters for the reference camera and the target camera in the 3D dataset, and determines the 3D dataset based on at least some of the set parameters. Extracts a reference image and a target image, generates an occlusion mask applied to the reference image by the target image based on at least some of the set parameters, and creates an overlap between the reference image and the target image based on at least some of the set parameters. You can create warp vectors from mask and target images to reference images.

Also, the image dataset generating apparatus 110 according to an embodiment may further include an interface capable of providing information to a user.

In the image dataset generation apparatus 110 shown in FIG. 5, only components related to the present embodiment are shown. Accordingly, those skilled in the art can understand that other general-purpose components may be further included in addition to the components shown in FIG. 5 .

The device according to the above-described embodiments includes a processor, a memory for storing and executing program data, a permanent storage unit such as a disk drive, a communication port for communicating with an external device, a touch panel, a key, a button, and the like. The same user interface device and the like may be included. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on the processor. Here, the computer-readable recording medium includes magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM) ), and DVD (Digital Versatile Disc). A computer-readable recording medium may be distributed among computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed by a processor.

This embodiment can be presented as functional block structures and various processing steps. These functional blocks may be implemented with any number of hardware or/and software components that perform specific functions. For example, an embodiment is an integrated circuit configuration such as memory, processing, logic, look-up table, etc., which can execute various functions by control of one or more microprocessors or other control devices. can employ them. Similar to components that can be implemented as software programming or software elements, the present embodiments include data structures, processes, routines, or various algorithms implemented as combinations of other programming constructs, such as C, C++, Java ( It can be implemented in a programming or scripting language such as Java), assembler, or the like. Functional aspects may be implemented in an algorithm running on one or more processors. In addition, this embodiment may employ conventional techniques for electronic environment setting, signal processing, message processing, and/or data processing. Terms such as "mechanism", "element", "means", and "composition" may be used broadly and are not limited to mechanical and physical components. The term may include a meaning of a series of software routines in association with a processor or the like.

The foregoing embodiments are merely examples and other embodiments may be implemented within the scope of the claims described below.

Claims (15)

In the method of generating an image dataset for learning and evaluating a model,
setting one or more parameters for a reference camera and a target camera in a 3D dataset;
extracting a reference image and a target image from the 3D dataset based on at least some of the set parameters;
generating an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters; and
generating an overlap mask between the reference image and the target image and a warp vector from the target image to the reference image based on at least some of the set parameters; How to create a dataset. According to claim 1,
The one or more parameters are
Position coordinates of the reference camera, position coordinates of the target camera, Euler angle of the reference camera, Euler angle of the target camera, focal length of the reference camera, focal length of the target camera, A method of generating an image dataset, comprising at least a part of a resolution, a resolution of the target camera, a projection matrix of the reference camera, and a projection matrix of the target camera. According to claim 1,
The extraction step is
The method of generating an image dataset, characterized in that the reference image is extracted by adjusting the positional coordinates and Euler angles of the reference camera, and the target image is extracted by adjusting the positional coordinates and Euler angles of the target camera. According to claim 3,
The extraction step is
The position coordinates of the reference camera and Euler angles are adjusted by scaling and summing randomly extracted unit vectors within a preset range to each set value of the position coordinates of the reference camera and Euler angles, and the position of the target camera is adjusted. A method of generating an image dataset, characterized by adjusting the position coordinates of the target camera and the Euler angles by scaling and adding randomly extracted unit vectors within a preset range to each set value of the coordinates and Euler angles. According to claim 1,
The step of generating the occlusion mask,
Based on a ray casting algorithm, the occlusion mask is generated by projecting a first ray at the position coordinates of the reference camera and projecting a second ray at the position coordinates of the target camera, characterized in that the image data How to create a set. According to claim 5,
The step of generating the occlusion mask,
projecting the first light ray in a direction indicated by each pixel of the reference image from the location coordinates of the reference camera;
calculating the coordinates of an object that the first ray first encounters in a traveling path as first coordinates;
projecting the second light ray toward the first coordinates from the location coordinates of the target camera;
calculating the coordinates of an object that the second ray encounters first in a traveling path as second coordinates; and
and generating the occlusion mask based on whether the first coordinates match the second coordinates. According to claim 6,
The step of generating the occlusion mask,
assigning 1 to a pixel of the occlusion mask corresponding to a pixel of the reference image when the first coordinate and the second coordinate coincide;
and allocating 0 to a pixel of the occlusion mask corresponding to a pixel of the reference image when the first coordinate and the second coordinate do not match. According to claim 1,
Generating the overlap mask and the warp vector,
and generating the overlapping mask and the warp vector based on the projection matrix of the reference camera and the projection matrix of the target camera. According to claim 8,
Generating the overlap mask and the warp vector,
Calculating an inverse matrix of the projection matrix of the reference camera;
calculating three-dimensional coordinates for each pixel of the reference image by matrix-multiplying the coordinates of each pixel of the reference image with the inverse matrix;
Calculating projection coordinates from the target camera to each pixel by matrix multiplication of the projection matrix of the target camera by the three-dimensional coordinates of each pixel of the reference image;
generating the overlapping mask based on whether each of the projection coordinates is included in the target image; and
and generating the warp vector based on the coordinates of each pixel of the reference image and the projection coordinates, respectively. According to claim 9,
Generating the overlapping mask,
If the projection coordinates are included in the target image, assigning 1 to pixels of the overlapping mask corresponding to pixels of the reference image;
and allocating 0 to pixels of the overlapping mask corresponding to pixels of the reference image when the projection coordinates are not included in the target image. According to claim 9,
Generating the warp vector,
and generating the warp vector by subtracting each of the projected coordinates from the coordinates of each pixel of the reference image. According to claim 1,
The occlusion mask, the overlap mask, and the warp vector,
Image data characterized in that it is used as a ground truth (GT) in the process of training and evaluation of a deep learning model that performs image stitching on the reference image and the target image How to create a set. one or more processors;
Memory; and
contains one or more programs;
the one or more programs are stored in the memory and configured to be executed by the one or more processors;
The one or more programs,
instructions for setting one or more parameters for a reference camera and a target camera in a 3D dataset;
a command for extracting a reference image and a target image from the 3D dataset based on at least some of the set parameters;
an instruction for generating an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters; and
Based on at least some of the set parameters, generating an overlap mask between the reference image and the target image and a warp vector from the target image to the reference image, Device. combined with hardware
setting one or more parameters for a reference camera and a target camera in a 3D dataset;
extracting a reference image and a target image from the 3D dataset based on at least some of the set parameters;
generating an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters; and
Based on at least some of the set parameters, a computer to execute the step of generating an overlap mask between the reference image and the target image and a warp vector from the target image to the reference image. A computer program stored on a readable recording medium. As a non-transitory computer readable storage medium,
a medium configured to store computer readable instructions;
The computer readable instructions, when executed by a processor, cause the processor to:
setting one or more parameters for a reference camera and a target camera in a 3D dataset;
extracting a reference image and a target image from the 3D dataset based on at least some of the set parameters;
generating an occlusion mask applied to the reference image by the target image, based on at least some of the set parameters; and
generating an overlap mask between the reference image and the target image and a warp vector from the target image to the reference image, based on at least some of the set parameters; A non-transitory computer-readable storage medium that enables performing a method of generating a set.

Images