KR102620875B1 - Method and device for dcnn-based image stitching - Google Patents

Abstract

A deep neural network-based image stitching method and device are disclosed. An image stitching method according to an embodiment includes receiving a plurality of images from different viewpoints, obtaining image stitching parameters based on the plurality of images, and creating a warped panoramic image based on the image stitching parameters. It may include an operation of generating, an operation of generating a parallax distortion-removed image from the warped panoramic image, and an operation of generating a stitched panoramic image by stitching the plurality of images from the parallax distortion-removed image.

Description

Deep neural network-based image stitching method and device {METHOD AND DEVICE FOR DCNN-BASED IMAGE STITCHING}

The disclosure below relates to a deep neural network-based image stitching method and apparatus.

Image stitching technology is a technology that creates one image by naturally connecting two or more images with overlapping parts. Image stitching technology is implemented through methods such as feature point extraction, feature point matching in overlapping areas, homography estimation, warping, and blending.

Typical image stitching techniques include extracting and matching key points and descriptors, estimating extrinsic parameters, and warping/matching to the target image. It is implemented through registration into a common target image, blending or seamline removal steps.

Recent non-deep-learning based stitching technology uses SIFT (Scale-Invariant Feature Transform) to improve the step of finding and matching image keypoints. Conventional methods are designed for image pairs with large parallax and large overlapping areas, and use homography to handle large parallax.

Recent deep learning-based stitching technology has replaced SIFT-based keypoint extraction with CNN (Convolutional Neural Network), but it does not consider lens distortion and cannot handle parallax distortion after warping.

According to various embodiments, it is possible to provide a technology for receiving a plurality of images from different viewpoints and generating a stitched panoramic image by stitching the plurality of images.

According to various embodiments, image stitching may be performed based on image stitching parameters obtained from a plurality of images.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

An image stitching method according to an embodiment includes receiving a plurality of images from different viewpoints, obtaining image stitching parameters based on the plurality of images, and stitching the plurality of images based on the image stitching parameters. The operation of generating a stitched panoramic image may be included.

The image stitching parameters may include lens distortion removal parameters, warping parameters, and stitching vectors.

Generating the stitched panoramic image may include generating a plurality of lens distortion removal images from each of the plurality of images based on the lens distortion removal parameter.

Generating the stitched panoramic image may further include generating a warped panoramic image from the plurality of lens distortion-removed images based on the warping parameter.

Generating the stitched panoramic image may further include generating a parallax distortion-free image from the warped panoramic image based on the stitching vector.

The operation of generating the parallax distortion-free image includes setting a stitching seam of the warped panoramic image, setting a stitching point at the stitching seam, and creating the warped panoramic image based on the stitching point and the stitching vector. It may include an operation of generating the parallax distortion-removed image from .

The operation of generating the stitched panoramic image includes obtaining a residual stitching vector based on the stitching point and the stitching vector and performing pixel alignment of the parallax distortion-removed image based on the residual stitching vector. It may further include an operation to generate a .

Obtaining image stitching parameters based on the plurality of images may include obtaining image stitching parameters by inputting the plurality of images into a learned neural network.

The learned neural network is supervised to minimize a loss function based on the correct answer image and the stitched panoramic image and a loss function based on the warped panoramic image, and self-supervised learning is performed to minimize a loss function based on the warped panoramic image. It may be a neural network.

An image stitching device according to an embodiment includes a main memory for storing one or more instructions and a processor for executing the instructions, and when the instructions are executed, the processor displays a plurality of images from different view angles. An image may be received, image stitching parameters may be obtained based on the plurality of images, and a stitched panoramic image may be generated by stitching the plurality of images based on the image stitching parameters.

The image stitching parameters may include lens distortion removal parameters, warping parameters, and stitching vectors.

The processor may generate a plurality of lens distortion removal images from each of the plurality of images based on the lens distortion removal parameter.

The processor may generate a warped panoramic image from the plurality of lens distortion-removed images based on the warping parameter.

The processor may generate a parallax distortion-free image from the warped panoramic image based on the stitching vector.

The processor sets a stitching seam of the warped panoramic image, sets a stitching point at the stitching seam, and generates the parallax distortion-free image from the warped panoramic image based on the stitching point and the stitching vector. You can.

The processor may generate the stitched panoramic image by obtaining a residual stitching vector based on the stitching point and the stitching vector, and performing pixel alignment of the parallax distortion-free image based on the residual stitching vector.

The processor may obtain image stitching parameters by inputting the plurality of images into a learned neural network.

The learned neural network is supervised to minimize a loss function based on the correct answer image and the stitched panoramic image and a loss function based on the warped panoramic image, and self-supervised learning is performed to minimize a loss function based on the warped panoramic image. It may be a neural network.

1 is a schematic block diagram of an image stitching device according to an embodiment.
Figure 2 is a diagram for explaining an operation for removing lens distortion of an input image.
Figure 3 is a diagram for explaining the operation of generating a warped panoramic image from a lens distortion-removed image.
Figure 4 is a diagram for explaining an operation for removing parallax distortion of a warped panoramic image.
Figure 5 is a diagram for explaining the pixel alignment operation of an image from which parallax distortion is removed.
Figure 6 is an example of a plurality of images from different viewpoints and a stitched panoramic image.
FIG. 7 is an example of a stitched panoramic image output by the image stitching device shown in FIG. 1 and an image generated by a conventional stitching method.
Figure 8 is a flowchart of an image stitching method according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a schematic block diagram of an image stitching device according to an embodiment.

The image stitching device 100 may receive a plurality of images from different viewpoints and generate a stitched panoramic image by stitching the plurality of images. The image stitching device 100 may perform image stitching based on image stitching parameters obtained from a plurality of images. The image stitching apparatus 100 may obtain image stitching parameters from a learned neural network, and the neural network may be learned using a semi-supervised learning method. The image stitching device 100 may simultaneously perform distortion removal (e.g., lens distortion removal, parallax distortion removal), and image stitching based on image stitching parameters. Image stitching device 100 ) can create a panoramic image with lens distortion and parallax distortion removed, and can create a high-resolution panoramic image through a pixel level alignment process.

The image stitching device 100 can perform neural network operations and learn neural networks. The image stitching device 100 may perform inference based on a learned neural network.

Neural networks (or artificial neural networks) can include statistical learning algorithms that mimic neurons in biology in machine learning and cognitive science. A neural network can refer to an overall model in which artificial neurons (nodes), which form a network through the combination of synapses, change the strength of the synapse connection through learning and have problem-solving capabilities.

Neurons in a neural network can contain combinations of weights or biases. A neural network may include one or more layers consisting of one or more neurons or nodes. Neural networks can infer the results they want to predict from arbitrary inputs by changing the weights of neurons through learning.

Neural networks may include deep neural networks. Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), and LSTM. (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield) Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network) ), Liquid State Machine (LSM), Extreme Learning Machine (ELM), Echo State Network (ESN), Deep Residual Network (DRN), Differential Neural Computer (DNC), Neural Turning Machine (NTM), Capsule Network (CN), It may include Kohonen Network (KN) and Attention Network (AN).

The image stitching device 100 may be implemented with a printed circuit board (PCB) such as a motherboard, an integrated circuit (IC), or a system on chip (SoC). For example, the image stitching device 100 may be implemented with an application processor.

Additionally, the image stitching device 100 may be implemented in a personal computer (PC), a data server, or a portable device. Portable devices include laptop computers, mobile phones, smart phones, tablet PCs, mobile internet devices (MIDs), personal digital assistants (PDAs), and enterprise digital assistants (EDAs). , digital still camera, digital video camera, portable multimedia player (PMP), personal navigation device or portable navigation device (PND), handheld game console, e-book ( It can be implemented as an e-book) or a smart device. A smart device may be implemented as a smart watch, smart band, or smart ring.

The image stitching device 100 may include a memory 110 and a processor 130.

The memory 110 may store instructions (or programs) executable by the processor 130. For example, the instructions may include instructions for executing the operation of the processor and/or the operation of each component of the processor.

The memory 110 may be implemented as a volatile memory device or a non-volatile memory device.

Volatile memory devices may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

Non-volatile memory devices include EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM (MRAM), and Conductive Bridging RAM (CBRAM). , FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM (Nanotube RRAM), Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, molecular electronic memory device, or insulation resistance change memory.

The processor 130 may process data stored in the memory 110. The processor 130 may execute computer-readable code (eg, software) stored in the memory 110 and instructions triggered by the processor 130 .

The processor 130 may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the intended operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

The processor 130 receives a plurality of images from different viewpoints, acquires image stitching parameters (e.g., lens distortion removal parameters, warping parameters, and stitching vectors) based on the plurality of images, and stores the image stitching parameters based on the obtained image stitching parameters. Based on this, a stitched panoramic image can be created by stitching a plurality of images.

The processor 130 generates a plurality of lens distortion-removed images from each of the plurality of images based on the lens distortion removal parameter, generates a warped panoramic image from the plurality of lens distortion-removal images based on the warping parameter, and adds a warped panoramic image to the stitching vector. A parallax distortion-removed image can be generated from a warped panoramic image based on the warped panoramic image, and a stitched panoramic image can be generated by performing pixel alignment of the parallax distortion-removed image based on the residual stitching vector.

The processor 130 may train a neural network that outputs image stitching parameters (eg, lens distortion removal parameters, warping parameters, and stitching vectors). The processor 130 can learn a neural network through semi-supervised learning. The processor 130 can learn a neural network by first performing supervised learning and then performing self-supervised learning.

The processor 130 can supervised learning of a neural network to minimize the loss function based on the correct answer image and the stitched panoramic image and the loss function based on the warped panoramic image.

Supervised learning is a method of inputting input data and corresponding output data together into a neural network and updating the connection weights of the connection lines so that output data corresponding to the input data is output.

For example, the processor 130 may update connection weights between artificial neurons through delta rule and error backpropagation learning.

Error backpropagation learning estimates the error through forward computation for given learning data, then starts from the output layer and moves backwards toward the hidden layer and input layer to propagate the estimated error and reduce the error. This is a method of updating connection weights in each direction.

Neural network processing proceeds in the direction of the input layer, hidden layer, and output layer, but in error backpropagation learning, the update direction of the connection weight may proceed in the direction of the output layer, hidden layer, and input layer.

The processor 130 defines an objective function to measure how close the currently set connection weights are to the optimal, continues to change the connection weights based on the results of the objective function, and repeatedly performs learning. .

For example, the objective function may be a loss function for calculating the error between the output value actually output by the neural network based on learning data and the expected value desired to be output. The processor 130 may update the connection weights in a direction that reduces the value of the loss function.

The processor 130 may calculate the first loss function through Equation 1 and perform supervised learning of the neural network to minimize the first loss function value.

[Equation 1]

In equation 1, is the loss function based on the correct answer image and the stitched panoramic image, is a loss function based on the overlap area of the warped panoramic image, is a stitched panoramic image, is the correct answer video, is the warped left image, is the warped right image, is the stitched panoramic image at time t, is the stitched panoramic image at time t+1, may be a threshold.

Locally moving objects in the image may cause inconsistencies in the time axis, and the processor 130 determines the threshold value. Based on , locally moving objects can be excluded from the loss function calculation by masking them. threshold may be set to 0.02, for example.

After performing supervised learning of the neural network, the processor 130 may perform semi-supervised learning of the neural network based on the second loss function calculated through Equation 2.

[Equation 2]

In equation 2, is the warped left image, may be the warped right image. The processor 130 can generate a high-resolution stitched panoramic image by training a neural network to minimize the second loss function.

The processor 130 can perform semi-supervised learning of a neural network by first performing supervised learning and then performing self-supervised learning.

The processor 130 may obtain image stitching parameters (e.g., lens distortion removal parameters, warping parameters, and stitching vectors) by inputting a plurality of images from different viewpoints (e.g., left image, right image) into the learned neural network. .

Hereinafter, the operation of generating a stitched panoramic image by stitching the input left image and the input right image based on the acquired image stitching parameters will be described. However, the input image is not limited to the input left image and the input right image, and a stitched panoramic image can be created by stitching three or more images.

Figure 2 is a diagram for explaining an operation for removing lens distortion of an input image.

Figure 2(A) shows an input image, and Figure 2(B) shows a lens distortion-removed image obtained from the input image.

The processor (processor 130 in FIG. 1) removes lens distortion removal parameters (e.g., left image, right image) corresponding to each of a plurality of input images (e.g., left image, right image) from the learned neural network. , , , ) can be obtained, and the processor 130 can obtain each lens distortion removal parameter ( , , , )), a plurality of images with lens distortion removed (e.g., left image with lens distortion removed, right image with lens distortion removed) can be generated.

The processor 130 can generate a lens distortion-removed image by matching the coordinates of the lens-distortion-removed image with the coordinates of the input image through Equation 3.

[Equation 3]

In Equation 3, ( , ) are the coordinates of the lens distortion-removed image, ( , ) are the coordinates of the center pixel of the input image, ( , ) are the coordinates of the input image, and is the lens distortion removal parameter, Is ( , )and ( , ) may be the Euclidean distance.

Lens distortion may follow a radial model in which the degree becomes worse as the distance from the center of the image increases, and the lens distortion-removed image may have a concave shape compared to the input image. The processor 130 can generate a stitched panoramic image that is robust to lens distortion by removing lens distortion of the input image before performing image stitching.

Figure 3 is a diagram for explaining the operation of generating a warped panoramic image from a lens distortion-removed image.

The processor (processor 130 in FIG. 1) may generate the left side 301 of the warped panoramic image based on the left image with lens distortion removed, and the right side 302 of the warped panoramic image based on the right image with lens distortion removed. ) can be created. The processor 130 may generate one warped panoramic image (FIG. 3) from the left side 301 of the warped panoramic image and the right side 302 of the warped panoramic image.

The processor 130 generates warping parameters (e.g., left image, right image) corresponding to each of a plurality of input images (e.g., left image, right image) from the learned neural network. , , , , ) can be obtained, and the processor 130 can generate a warped panoramic image (FIG. 3) based on each warping parameter.

The processor 130 can generate the left side 301 of the warped panoramic image by matching the coordinates of the warped panoramic image with the coordinates of the left image from which lens distortion is removed using Equation 4.

[Equation 4]

In Equation 4, ( , ) are the coordinates of the left image without lens distortion, ( , ) are the coordinates of the warped panoramic image, and is the focal length of the input left image, W is the width of the input left image, H is the height of the input left image, is the Euler angle of the input left image, and is the focal length of the warped panoramic image, ( , ) may be the coordinates of the center pixel (or the coordinates of key points) of the warped panoramic image.

The processor 130 can generate the right side 302 of the warped panoramic image by matching the coordinates of the warped panoramic image with the coordinates of the right image from which lens distortion is removed using Equation 5.

[Equation 5]

Since the parameters of Equation 5 correspond to the parameters of Equation 4, detailed description will be omitted.

The processor 130 sets the warping parameters ( , ), the lens distortion-removed image can be rotated in three dimensions based on the warping parameter ( , , ), the viewpoint (or focus) of the lens distortion removal image can be moved to the viewpoint (or viewpoint) of the panoramic image. Processor 130 may configure warping parameters, e.g. , , , , ), the vertical alignment and horizontal alignment of the image can be performed.

The processor 130 may generate the left side 301 of the warped panoramic image based on the left image with lens distortion removed, and the right side 302 of the warped panoramic image based on the right image with lens distortion removed. . The processor 130 can generate one warped panoramic image (FIG. 3) from the left 301 of the warped panoramic image and the right 302 of the warped panoramic image, and the warped panoramic image (FIG. 3) is It may include an overlapping area 303. Hereinafter, the operation of removing the overlapping area 303 will be described.

Figure 4 is a diagram for explaining an operation for removing parallax distortion of a warped panoramic image.

Figure 4(A) shows a portion of the warped panoramic image of Figure 3, and Figure 4(B) shows a parallax distortion-free image generated from the warped panoramic image.

The overlap area 303 may be caused by parallax distortion based on the difference in distance between the camera and the target, or may be caused by ghosting based on the difference in pose of the camera. The processor (processor 130 in FIG. 1) may generate stitching vectors, e.g. and ) The overlapping area 303 can be removed based on.

The processor 130 generates a stitching vector (e.g., a left image, a right image) from a plurality of input images (e.g., left image, right image) from the learned neural network. and ) can be obtained, and the processor 130 can generate a parallax distortion-free image based on the stitching vector.

The processor 130 may set a stitching seam of the warped panoramic image (FIG. 3), and the stitching seam may be a center line portion of the warped panoramic image. Processor 130 may identify stitching points at a stitching seam, e.g. and ), and set the stitching point ( and ) and stitching vectors (e.g. and ), a parallax distortion-removed image (FIG. 4(B)) can be generated from the warped panoramic image (FIG. 4(A)). Processor 130 is a stitching point ( and ) and stitching vectors (e.g. and ), the overlapping area 303 can be removed by pulling or pushing the left side 301 of the warped panoramic image and the right side 302 of the warped panoramic image. The degree to which a pixel is pulled or pushed depends on the stitching point, e.g. and ) and can increase or decrease linearly based on the distance between pixels.

Processor 130 may select a stitching vector, e.g. and ), a parallax distortion-free image can be generated from the warped panoramic image.

Figure 5 is a diagram for explaining the pixel alignment operation of an image from which parallax distortion is removed.

Figure 5(A) shows an image before pixel alignment, and Figure 5(B) shows an image after pixel alignment.

The processor (processor 130 in FIG. 1) may select a stitching point (e.g. and ) and stitching vectors (e.g. and ) Based on the residual stitching vector ( and ) can be obtained. Specifically, processor 130 selects a stitching point (e.g. and ) defined based on Navigate the pixel window to find the residual stitching vector ( and ) can be obtained.

The processor 130 generates a residual stitching vector ( and ), a stitched panoramic image can be created by performing pixel alignment of the parallax distortion-removed image. The processor 130 generates a residual stitching vector ( and ), pixel alignment of the parallax distortion-removed image can be performed and a stitched panoramic image can be generated.

FIG. 6 is an example of a plurality of images from different viewpoints and a stitched panoramic image, and FIG. 7 is an example of a stitched panoramic image output by the image stitching device shown in FIG. 1 and an image generated by a conventional stitching method.

Figure 6(A) shows the input left image, Figure 6(B) shows the input right image, and Figure 6(C) shows the output panoramic image.

The image stitching device (image stitching device 100 in FIG. 1) receives a plurality of images (e.g., (A) in FIG. 6 and (B) in FIG. 6) from different viewpoints and creates a plurality of images (in FIG. 6). A stitched panoramic image (e.g., (C) of FIG. 6) obtained by stitching (A) and (B) of FIG. 6) can be generated and output. The image stitching device 100 may perform image stitching based on image stitching parameters obtained from a plurality of images (FIG. 6(A) and FIG. 6(B)). The image stitching apparatus 100 may obtain image stitching parameters from a learned neural network, and the neural network may be learned using a semi-supervised learning method.

Figure 7(A) shows a high-resolution image generated by the image stitching device 100, and Figure 7(B) shows a low-resolution image generated by a conventional stitching method.

The image stitching device 100 may simultaneously perform distortion removal (e.g., lens distortion removal, parallax distortion removal) and image stitching based on image stitching parameters. The image stitching device 100 can generate a panoramic image with lens distortion and parallax distortion removed, and can generate a high-resolution panoramic image through a pixel level alignment process.

Figure 8 is a flowchart of an image stitching method according to an embodiment.

Referring to FIG. 8, an image stitching device (image stitching device 100 of FIG. 1) may perform an image stitching method.

In operation 801, the image stitching device 100 may receive a plurality of images from different viewpoints.

In operation 802, the image stitching device 100 may obtain image stitching parameters based on a plurality of images.

In operation 803, the image stitching device 100 may generate a stitched panoramic image by stitching a plurality of images based on image stitching parameters.

Since the image stitching method performed by the image stitching device 100 is substantially the same as described above, detailed description will be omitted.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (9)

An operation of receiving a plurality of images from different viewpoints;
Obtaining image stitching parameters based on the plurality of images;
generating a warped panoramic image based on the image stitching parameters;
generating a parallax distortion-removed image from the warped panoramic image; and
An operation of generating a stitched panoramic image by stitching the plurality of images from the parallax distortion-removed image,
The video stitching parameters are,
Lens distortion removal parameters, warping parameters, and stitching vectors
Including,
The operation of acquiring image stitching parameters based on the plurality of images includes:
An operation of acquiring the image stitching parameters by inputting the plurality of images into a learned neural network,
The learned neural network is,
Supervised learning is performed to minimize a loss function based on the correct answer image and the stitched panoramic image and a loss function based on the warped panoramic image,
An image stitching method, which is a self-supervised learning neural network to minimize a loss function based on the warped panoramic image.
delete According to paragraph 1,
The operation of generating the warped panoramic image is,
An operation of generating a plurality of lens distortion removal images from each of the plurality of images based on the lens distortion removal parameter.
Including, video stitching method.
According to paragraph 3,
The operation of generating the warped panoramic image is,
An operation of generating the warped panoramic image from the plurality of lens distortion-removed images based on the warping parameter.
A video stitching method further comprising:
According to paragraph 4,
The operation of generating the parallax distortion-removed image is,
An operation of generating the parallax distortion-removed image from the warped panoramic image based on the stitching vector.
A video stitching method further comprising:
According to clause 5,
The operation of generating the parallax distortion-removed image from the warped panoramic image based on the stitching vector includes:
Setting a stitching seam of the warped panoramic image;
An operation of setting a stitching point at the stitching seam; and
An operation of generating the parallax distortion-removed image from the warped panoramic image based on the stitching point and the stitching vector.
Including, video stitching method.
According to clause 6,
The operation of generating the stitched panoramic image is,
Obtaining a residual stitching vector based on the stitching point and the stitching vector; and
An operation of generating the stitched panoramic image by performing pixel alignment of the parallax distortion-removed image based on the residual stitching vector.
A video stitching method further comprising:
delete delete