KR102442980B1 - Super-resolution method for multi-view 360-degree image based on equi-rectangular projection and image processing apparatus - Google Patents

Abstract

The ERP-based super-resolution method of multi-view 360-degree images includes the steps of: receiving, by an image processing device, a low-resolution target image and a high-resolution reference image among a plurality of 360-degree images that provide multi-view, the image processing device is the target image and inputting the reference image to a disparity estimation model and outputting disparity information of the target image and the reference image, aligning the reference image based on the disparity information by the image processing apparatus, the image and outputting, by a processing unit, a high-resolution image of the target image by inputting the target image and the aligned reference image to a reconstruction layer composed of residual blocks. The target image and the reference image are images modeled as a 360-degree image based on an ERP (Equirectangular Projection).

Description

SUPER-RESOLUTION METHOD FOR MULTI-VIEW 360-DEGREE IMAGE BASED ON EQUI-RECTANGULAR PROJECTION AND IMAGE PROCESSING APPARATUS

The technology to be described below relates to a super-resolution technique for multi-view 360-degree images based on ERP.

With the development of multimedia technology, interest in immersive media is increasing recently. The 360-degree image provides an omnidirectional image like the real physical environment based on a specific point of view. In addition, the multi-view image refers to an image acquired by a plurality of cameras having different viewpoints. The multi-view 360-degree image refers to an image acquired by a plurality of cameras that capture a 360-degree image. There are various techniques for mapping a two-dimensional image to a 360-degree image. ERP (Equi-rectangular projection) applying the cylindrical projection method is a representative mapping technique.

Various techniques are being studied for image super-resolution. There is also a super-resolution technique using a reference image. When there is an image associated with an image of a specific viewpoint, such as a 360-degree image or a multi-view 360-degree image, super-resolution using a reference image is possible.

US Patent Publication No. US 2013-0258048

In the reference image-based super-resolution technique, it is important to extract information that the low-resolution image can refer to from the reference image by exploring the correspondence between the input images. However, the conventional deep learning model has difficulty in compensating for disparity between images when the viewpoint difference is large, and cannot handle nonlinear distortion according to the latitude of the ERP image.

The technology described below intends to provide a super-resolution technique for multi-view 360-degree images based on ERP.

The ERP-based super-resolution method of multi-view 360-degree images includes the steps of: receiving, by an image processing device, a low-resolution target image and a high-resolution reference image among a plurality of 360-degree images that provide multi-view, the image processing device is the target image and inputting the reference image to a disparity estimation model and outputting disparity information of the target image and the reference image, aligning the reference image based on the disparity information by the image processing apparatus, and the image and outputting, by a processing unit, a high-resolution image of the target image by inputting the target image and the aligned reference image to a reconstruction layer composed of residual blocks.

The RP-based image processing device for super-resolution 360-degree images is an input device that receives a low-resolution target image and a high-resolution reference image among a plurality of 360-degree images that provide multiple views, and a 360-degree image in consideration of distortion between ERP images. A storage device for storing a neural network model for super-resolution and inputting the target image and the reference image to the neural network model to generate disparity information of the target image and the reference image, and the target image and the disparity information and a computing device generating a high-resolution image of the target image by using the reference image aligned with reference to .

The technique to be described below aligns a reference image to a super-resolution target image based on disparity between input images to perform super-resolution robust against ERP distortion.

1 is an example of a multi-view 360-degree imaging system.
2 is an example of a neural network model that performs super-resolution.
3 is another example of a neural network model that performs super-resolution.
4 is an example of a 360 degree disparity estimator of a pyramid structure.
5 is an example of an image processing apparatus that performs super-resolution.

Since the technology to be described below can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, and B may be used to describe various components, but the components are not limited by the above terms, and only for the purpose of distinguishing one component from other components. used only as For example, a first component may be named as a second component, and similarly, the second component may also be referred to as a first component without departing from the scope of the technology to be described below. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In terms of terms used herein, the singular expression should be understood to include the plural expression unless the context clearly dictates otherwise, and terms such as "comprises" include the described feature, number, step, operation, element. , parts or combinations thereof are to be understood, but not to exclude the possibility of the presence or addition of one or more other features or numbers, step operation components, parts or combinations thereof.

Prior to a detailed description of the drawings, it is intended to clarify that the classification of the constituent parts in the present specification is merely a division according to the main function that each constituent unit is responsible for. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each more subdivided function. In addition, each of the constituent units to be described below may additionally perform some or all of the functions of other constituent units in addition to the main function it is responsible for. Of course, it can also be performed by being dedicated to it.

In addition, in performing the method or operation method, each process constituting the method may occur differently from the specified order unless a specific order is clearly described in context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The 360-degree image refers to video content that provides an image of a 360-degree view based on one point. One 360-degree image is provided through an image acquired through a 360-degree camera disposed at a point. There are various techniques for mapping a two-dimensional image to a 360-degree image. ERP using cylindrical projection is a representative mapping technique.

The multi-view 360-degree image is provided through an image acquired through a plurality of 360-degree cameras disposed in a predetermined area. The multi-view 360-degree image provides an image from a free viewpoint to a moving user by matching and synthesizing images acquired in a certain area.

The low-resolution image refers to an image whose resolution is lower than a predetermined reference value. A 360-degree low-resolution image is an image acquired by a 360-degree camera and has a low resolution. The 360-degree low-resolution image may be an image obtained by downscaling an image acquired by a 360-degree camera. Hereinafter, the low resolution image is referred to as a low resolution (LR) image.

The high-resolution image refers to an image having a higher resolution than a reference value. A 360-degree high-resolution image is an image acquired by a 360-degree camera and is an image with high resolution. The 360-degree high-resolution image refers to an original image acquired by a 360-degree camera or an image having a resolution greater than or equal to a reference value. Hereinafter, the high-resolution image is referred to as a high-resolution (HR) image.

Super-resolution refers to a technique for converting a low-resolution image into a high-resolution image. Conventionally, super-resolution was performed using a technique such as pixel interpolation. The technique to be described below is a super-resolution technique for a multi-view 360 degree image as described above. Hereinafter, a super-resolution technique for a multi-view 360-degree image is more simply expressed as MV-SR (multi-view super-resolution).

The technique described below performs super-resolution using a neural network model. The neural network model includes various models such as Recurrent Neural Networks (RNN), feedforward neural network (FFNN), and convolutional neural network (CNN). In the following description, CNN will be mainly described, but the super-resolution technology is not implemented only with a specific neural network model.

1 is an example of a multi-view 360-degree imaging system. 1 is an example of a system for performing super-resolution in a decoder.

The encoder 30 receives images from the plurality of cameras 11 to 15 . The encoder 30 may encode an individual image according to a 360-degree image format. In this case, the encoder 30 may downscale some of the plurality of 360-degree images to a low-resolution image. The storage device 50 stores 360-degree images (video streams 11 to 15) composed of images acquired by the plurality of cameras 11 to 15 . It is assumed that the video stream 13 is a low-resolution video.

360-degree images (video streams 11 to 15) are transmitted to the receiving end through the network. The decoder 70 decodes the encoded image. The image processing apparatus 100 performs super-resolution using the decoded image. The image processing apparatus 100 converts the video stream 13 into a high-resolution image using the low-resolution image video stream 13 and other adjacent images (at least one of video streams 11, 12, 14, and 15). The storage device 90 may store 360-degree images (video streams 11 to 15) that are all high-resolution images.

At the receiving end, the decoder 70 and the image processing apparatus 100 are separately displayed. However, one device may perform decoding and super-resolution. On the other hand, encoding or decoding is not related to the super-resolution process to be described below. The image processing apparatus performs super-resolution based on only the low-resolution image and the high-resolution image to be referenced.

Hereinafter, it is assumed that an apparatus for performing super-resolution on a multi-view 360-degree image is an image processing apparatus. The image processing apparatus may be physically various types. For example, the image processing device may be a VR device, a PC, a server, a chipset in which a program is embedded, and the like. The image processing apparatus receives a plurality of 360-degree images and performs super-resolution.

ERP image includes certain distortion in a specific area of the image. Multi-view images are obtained from a plurality of cameras disposed at different positions. It is a plurality of images taken from the same point or area, and the degree of distortion of the ERP image varies depending on the location of the camera. Therefore, it is not easy to super-resolution an image acquired by one 360-degree camera based on an image acquired by another camera. Therefore, the following super-resolution process is proposed.

An image to be super-resolution among a plurality of 360-degree images is called a target image. The target image is a low-resolution image. The target image is super-resolution using a low-resolution target image and a high-resolution image acquired by a camera adjacent to the camera that captured the target image. In this case, a high-resolution image acquired by an adjacent camera is called a reference image. The reference image includes all or part of a region captured by the target image. Referring to FIG. 2 , if the target image is an image acquired by the camera 13, the reference image may be any one of cameras 11, 12, 14 and 15.

Hereinafter, an apparatus for super-resolution a target image is referred to as an image processing apparatus. The image processing apparatus is a computer device capable of processing image data. For example, the image processing apparatus may be a PC, a smart device, a server on a network, a TV setup box, a VR device, or the like.

2 is an example of a neural network model 200 that performs super-resolution. The neural network model 200 of FIG. 2 receives a 360-degree input image and outputs a 360-degree high-resolution image. The input image is a low-resolution target image E ^LR and a reference image E ^Ref . The output image is the target image E ^LR , and the result of super-resolution is the high-resolution image E ^SR .

2 is a model of super-resolution a real 360-degree image using a transfer layer. The first transition layer 211 receives E ^LR and outputs a synthesized 360-degree image feature t ^LR . The second transition layer 212 receives E ^Ref and outputs a synthesized 360-degree image feature t ^Ref .

Disparity estimator 220 is E ^LR and E ^Ref extract the correlation between Correlation includes information on aligning related features among features extracted from each image.

The disparity estimator 220 calculates the correlation D _r→l and transfers the features of the reference image together with the low-resolution image without flow supervision.

The disparity estimator 220 is configured to extract a first feature from t ^LR of a target image, a configuration for extracting a second feature from t ^Ref of a reference image, and to estimate a flow from the correlation between the first feature and the second feature includes configuration.

Briefly, the first encoder 221 receives t ^LR and extracts f ^LR as a basic feature, and the first latitude-aware convolution 222 is a disc between f ^LR and ERP images. It serves to reduce the parity difference. The first location-aware convolutional layer 222 outputs s ^LR . The second encoder 223 receives t ^Ref and extracts f ^Ref as a basic feature, and the second location-aware convolutional layer 224 serves to reduce a disparity difference between ERP images in f ^Ref . The second location-aware convolutional layer 224 outputs s ^Ref . A double-headed arrow between the first encoder 221 and the second encoder 223, the first location-aware convolutional layer 222, and the second location-aware convolutional layer 224 indicates a relationship in which kernel parameters are shared.

로 와핑(warping)한다(240).

는 타깃 영상 E^LR과 참조 영상 E^Ref의 ERP 왜곡을 고려하여 참조 영상을 타깃 영상에 정렬한 결과에 해당한다.The image processing apparatus performs a correlation operation 225 between s ^LR and s ^Ref , and the flow estimator 226 receives the correlation information and finally E ^LR and E ^Ref . The correlation between D _r→l is output. The image processing device uses D _r→l to set E ^Ref .

and warping (240).

corresponds to the result of aligning the reference image with the target image in consideration of ERP distortion of the target image E ^LR and the reference image E ^Ref .

가 타깃과 관련하여 부정적인 특징(성능 저하 특징)을 갖는 경우, 초해상화 성능이 저하된다. 신경망 모델(200)은 이와 같은 성능 저하 특징을 제거하기 위한 구성을 더 포함할 수 있다. 마스크 생성기(230)는 t^LR 및 D_r→l을 입력받아

에 포함되는 성능 저하 특성을 제거하기 위한 마스크 M을 생성한다.

has a negative characteristic (performance degradation characteristic) with respect to the target, the super-resolution performance is degraded. The neural network model 200 may further include a configuration for removing such a performance degradation characteristic. Mask generator 230 is t ^LR and D _r→l as input

A mask M is created to remove the performance degradation characteristics included in .

을 요소별 곱셈(elementwise multiplication)하여 초해상화를 위한 최종적인 특징 데이터를 마련할 수 있다. The image processing device is M and

can be multiplied by element (elementwise multiplication) to prepare final feature data for super-resolution.

또는 M과

을 곱한 결과를 입력받아 영상을 재구성하여 초해상화를 수행한다. 재구성 계층(250)은 잔차 블록(residual block)으로 구성될 수 있다. 잔차 블록은 입력 특징이 들어와 일정 값이 출력되는 과정에서 학습 효율에 따라 컨볼루션을 수행하거나 수행하지 않는 과정을 선택적으로 제공한다.The reconstruction layer 250 includes (i) a low-resolution target image E ^LR (or t ^LR ) and (ii)

or M and

Super-resolution is performed by reconstructing the image by receiving the result of multiplying by . The reconstruction layer 250 may be composed of a residual block. The residual block selectively provides a process in which a convolution is performed or not performed according to learning efficiency in the process where an input feature is input and a predetermined value is output.

The third transition layer 260 converts the super-resolution synthetic 360-degree image output from the reconstruction layer 250 into a photorealistic synthetic image. The third transition layer 260 outputs a super-resolution 360 image E ^LR . In the neural network model 200 , the transition layers 211 , 212 , and 260 perform super-resolution of a high-complexity photorealistic image with low complexity in a synthetic image dimension.

3 is another example of a neural network model 300 that performs super-resolution. 3 is a super-resolution model for a synthetic 360-degree image.

The input image is a composite image, and is a low-resolution target image t ^LR and a reference image t ^Ref . The output image is the target image t ^LR , and the result of super-resolution is the high-resolution image t ^SR .

The disparity estimator 310 has t ^LR and t ^Ref extract the correlation between Correlation includes information on aligning related features among features extracted from each image.

The disparity estimator 310 calculates the correlation D _r→l and transfers the features of the reference image together with the low-resolution image without flow supervision.

The disparity estimator 310 is configured to extract a first feature from t ^LR of a target image, a configuration for extracting a second feature from t ^Ref of a reference image, and to estimate a flow from the correlation between the first feature and the second feature includes configuration.

Briefly, the first encoder 311 receives t ^LR and extracts f ^LR as a basic feature, and the first location-aware convolutional layer 312 serves to reduce the disparity difference between the ERP images in f ^LR . do The first location-aware convolutional layer 222 outputs s ^LR . The second encoder 313 receives t ^Ref and extracts f ^Ref as a basic feature, and the second location-aware convolutional layer 314 serves to reduce a disparity difference between ERP images in f ^Ref . The second location-aware convolutional layer 314 outputs s ^Ref . A double arrow between the first encoder 311 and the second encoder 313 , the first location-aware convolutional layer 312 and the second location-aware convolutional layer 314 indicates a relationship that shares a kernel parameter.

로 와핑(warping)한다(320).

는 타깃 영상 t^LR과 참조 영상 t^Ref의 ERP 왜곡을 고려하여 참조 영상을 타깃 영상에 정렬한 결과에 해당한다.The image processing apparatus performs a correlation operation 315 between s ^LR and s ^Ref , and the flow estimator 326 receives the correlation information and finally t ^LR and t ^Ref . The correlation between D _r→l is output. The image processing device calculates t ^Ref using D _r→l

and warping ( 320 ).

corresponds to the result of aligning the reference image with the target image in consideration of ERP distortion of the target image t ^LR and the reference image t ^Ref .

가 타깃과 관련하여 부정적인 특징(성능 저하 특징)을 갖는 경우, 초해상화 성능이 저하된다. 신경망 모델(300)은 이와 같은 성능 저하 특징을 제거하기 위한 구성을 더 포함할 수 있다. 마스크 생성기(330)는 t^LR 및 D_r→l을 입력받아

에 포함되는 성능 저하 특성을 제거하기 위한 마스크 M을 생성한다.

has a negative characteristic (performance degradation characteristic) with respect to the target, the super-resolution performance is degraded. The neural network model 300 may further include a configuration for removing such a performance degradation characteristic. Mask generator 330 is t ^LR and D _r→l as input

A mask M is created to remove the performance degradation characteristics included in .

을 요소별 곱셈하여 초해상화를 위한 최종적인 특징 데이터를 마련할 수 있다. The image processing device is M and

can be multiplied by elements to prepare final feature data for super-resolution.

또는 M과

을 곱한 결과를 입력받아 영상을 재구성하여 초해상화를 수행한다. 재구성 계층(340)은 잔차 블록으로 구성될 수 있다. 잔차 블록은 입력 특징이 들어와 일정 값이 출력되는 과정에서 학습 효율에 따라 컨볼루션을 수행하거나 수행하지 않는 과정을 선택적으로 제공한다. 재구성 계층(340)은 초해상화된 합성 360도 영상 t^SR을 출력한다.Reconstruction layer 340 is (i) t ^LR and (ii)

or M and

Super-resolution is performed by reconstructing the image by receiving the result of multiplying by . The reconstruction layer 340 may be composed of residual blocks. The residual block selectively provides a process in which a convolution is performed or not performed according to learning efficiency in the process where an input feature is input and a predetermined value is output. The reconstruction layer 340 outputs the super-resolution synthetic 360-degree image t ^SR .

A configuration prepared in consideration of the distortion between ERP images in the neural network model 200 of FIG. 2 and the neural network model 300 of FIG. 3 is a disparity estimator. A disparity estimator will be described. A location-aware convolutional layer (LatConv) is a key component for ERP distortion-resistant super-resolution.

In the ERP image, pixels located at a high latitude based on a spherical image are clustered in a longitudinal direction. Accordingly, in order to equalize the pixel density, it is necessary to horizontally expand the pixels from the equatorial region to the pole region. LatConv performs this operation in feature extraction of ERP images.

LatConv adaptively sets the sample interval of longitude according to latitude. The sampling interval a _i in the i-th row of the frame H(height)×W(width) may be defined as in Equation 1 below.

The image processing apparatus may horizontally scale the input feature map f according to latitude according to a _i . The image processing apparatus may perform (2K + 1) × (2K + 1) LatConv operation for each position (i, j) as shown in Equation 2 below.

Here, s is the output feature value, and w is the kernel. K may be set to 1 in the proposed network. LatConv can be applied across channels regardless of the dimension of the input feature f. LatConv can be trained as a standard reverse transcription. When the image processing apparatus applies the kernel to the boundary region of the image, it may perform circular padding in a horizontal direction and use a mirror symmetry scheme in a vertical direction.

A disparity estimator will be described. The encoders 221 , 223 , 311 , and 313 as a general convolutional layer extract f ^LR and f ^Ref . Because of the distortion between the two ERPs, f ^LR and f ^Ref It is difficult to find a reciprocal relationship between s ^LR and s ^Ref produced through LatConv The degree of ERP distortion becomes similar.

4 is an example of a 360 degree disparity estimator 400 of a pyramid structure.

The disparity estimator 400 includes an encoder 410 , a location-aware convolutional layer (LatConv, 420 ) and a flow estimator 440 .

The encoder 410 extracts features from each input image. The encoder 410 is composed of individual layers for extracting features for each of a target image and a reference image. Characteristics output from the same layer of the encoder 410 have different ERP distortions. For example, A and B have different spherical distortion.

The encoder 410 may extract pyramid-like features from a plurality of layers. The image processing apparatus may estimate disparity by comparing features of the same layer of the encoder 410 with respect to each of the target image and the reference image.

LatConv 420 performs a convolution operation on features in each layer of the pyramid. The LatConv 420 receives f _l ^LR and f _l ^Ref and outputs s _l ^LR and _sl ^Ref . l means the level of the hierarchy. The LatConv 420 aligns the features of the reference image with the features of the target image. The LatConv 420 matches the reference image to the target image by reducing distortion of feature density in longitude according to latitude. s _l ^LR and s _l ^Ref can be matched with spherical distortion. Distortion correction of the LatConv 420 is performed as in Equation (2).

에 정렬한다. The image processing apparatus matches the texture features between s _l ^LR and s _l ^Ref in each level l and inputs the calculated matching cost 430 to the flow estimator 440 to estimate the flow. The image processing apparatus combines the feature volume at the upper pyramid level and the estimated flow to estimate the flow at the current level. The flow estimator 440 estimates the disparity with a six-layer CNN. The image processing apparatus remaps the pixels to the corresponding coordinates to calculate the estimated disparity of E ^Ref .

sort on

, D_r→l 및 E^LR과

의 차이 절대값을 결합(concatenation)하여 입력하고, 결합마스크 M을 출력할 수 있다. 도 2 및 도 3에 도시한 바와 같이 마스크 생성기(230, 330)는 5개의 컨볼루션 계층 및 시그모이드 활성 계층으로 구성될 수 있다. 마스크 생성기(230, 330)는 영상 재구성(초해상화)을 위하여 어떤 영역이 사용되어야 하는지 결정한다. 참조 영상에서 바람직하지 않은 텍스처는 아래 수학식 3과 같이 억제(필터링)될 수 있다. 상기 수학식 3과 같은 데이터가 재구성 계층(250, 340)에 입력된다.Mask generators 230 and 330 are E ^LR ,

, D _r→l and E ^LR and

It is possible to input by concatenating the absolute difference value of , and output a concatenated mask M. 2 and 3 , the mask generators 230 and 330 may include five convolutional layers and a sigmoid active layer. The mask generators 230 and 330 determine which region should be used for image reconstruction (super-resolution). Undesirable textures in the reference image may be suppressed (filtered) as in Equation 3 below. Data as in Equation 3 is input to the reconstruction layers 250 and 340 .

The reconstruction layers 250 and 340 may be composed of a plurality of residual blocks having 64 filters.

The loss function used in the learning process will be described. Equations 4 and 5 below correspond to the loss function of the reconstruction layer and the loss function of the warping process of aligning the reference image to the target, respectively.

Here, E ^GT is the ground truth, and E ^SR is the super-resolution result. ρ may be set to 0.01. The overall loss function can be expressed as Equation 6 below.

5 is an example of an image processing apparatus 500 that performs super-resolution. The image processing apparatus 500 may be in the form of a VR device, a PC, a smart device, a network server, and the like.

The image processing apparatus 500 may include a storage device 510 , a memory 520 , an arithmetic device 530 , an interface device 540 , and a communication device 550 .

The storage device 510 may store programs or codes for the operation of the image processing apparatus 500 . The storage device 510 may store the aforementioned neural network model 200 or 300 . The storage device 510 may store a program or code for learning the neural network model 200 or 300 . The storage device 510 may store a high-resolution target image generated by the neural network model.

The memory 520 may temporarily store data and information generated during the operation of the image processing apparatus 500 .

The interface device 540 is a device that receives predetermined commands and data from the outside. The interface device 540 may receive predetermined information from a physically connected input device or a physical interface (keypad, touch panel, etc.). The interface device 540 may receive a neural network model, information for learning the neural network model, learning data, and the like. The interface device 540 may receive parameter values for updating the neural network model. The interface device 540 may receive a plurality of 360-degree images for super-resolution. The interface device 540 may receive the above-described target image and reference image.

The communication device 550 transmits and receives certain information through a wireless network. The communication device 550 may receive a neural network model, information for learning the neural network model, learning data, and the like. The communication device 550 may receive a parameter value for updating the neural network model. The communication device 550 may receive a target image and a reference image for inputting the neural network model. The communication device 550 may transmit a high-resolution target image generated by the neural network model to an external object.

The interface device 540 and the communication device 550 may receive certain information and data from a user or an external object. Accordingly, the interface device 540 and the communication device 550 may be collectively referred to as an input device.

The computing device 530 controls the operation of the image processing device 500 using a program or code stored in the storage device 510 . The computing unit 530 performs super-resolution using the neural network model.

The computing device 530 may be a device such as a processor, an AP, or a chip embedded with a program that processes data and processes a predetermined operation.

An operation of the computing device 530 will be described based on the neural network model 200 of FIG. 2 .

The calculating unit 530 generates a synthesized 360-degree image t ^LR by inputting the target image E ^LR that is the actual 360-degree image to the first transition layer 211 . The calculating unit 530 generates a synthetic 360-degree image t ^Ref by inputting the reference image E ^Ref , which is the actual 360-degree image, to the second transition layer 212 .

The arithmetic unit 530 inputs t ^LR to the first encoder 221 to output a feature value f ^LR , and inputs f ^LR to the first location-aware convolutional layer 222 to output s ^LR . The arithmetic unit 530 inputs t ^Ref to the second encoder 223 to output a feature value f ^Ref , and inputs f ^Ref to the second location-aware convolutional layer 224 to output s ^Ref .

arithmetic unit 530 is s ^LR and s ^Ref are compared in individual layers such as a pyramid structure as shown in FIG. 4 , and the correlation D _r→l is output using the flow estimation model 226 .

를 생성한다. The arithmetic unit 530 remaps (warps) the reference image E ^Ref based on the correlation D _r→l

create

을 입력하여 마스크 M을 생성한다.The operation unit 530 generates the target image E ^LR , the correlation D _r→l to the mask generation model or the mask generation layer 230 , and

to create a mask M.

과 마스크 M을 요소별로 곱셈한 결과를 재구성 계층(250)에 입력하여 합성 360도 고해상도 영상을 생성한다.The arithmetic unit 530 is

The result of multiplying the mask M and the mask M for each element is input to the reconstruction layer 250 to generate a synthesized 360-degree high-resolution image.

The calculating unit 530 generates an E ^SR that is a final live-action 360-degree image by inputting the output value of the reconstruction layer 250 to the third transition layer 260 .

An operation of the computing device 530 will be described based on the neural network model 300 of FIG. 3 .

The arithmetic unit 530 inputs a target image t ^LR that is a synthesized 360-degree image to the first encoder 311 to output a feature value f ^LR , and inputs f ^LR to the first location recognition convolution layer 312 to s ^LR is output. The arithmetic unit 530 inputs a reference image t ^Ref , which is a synthesized 360-degree image, to the second encoder 313 to output a feature value f ^Ref , and inputs f ^Ref to the second position recognition convolutional layer 314 to s Print ^Ref .

arithmetic unit 530 is s ^LR and s ^Ref are compared in individual layers such as a pyramid structure as shown in FIG. 4 , and the correlation D _r→l is output using the flow estimation model 326 .

를 생성한다. The arithmetic unit 530 remaps (warps) the reference image E ^Ref based on the correlation D _r→l

create

을 입력하여 마스크 M을 생성한다.The operation unit 530 applies the target image E ^LR , the correlation D _r→l to the mask generation model or the mask generation layer 330 , and

to create a mask M.

과 마스크 M을 요소별로 곱셈한 결과를 재구성 계층(340)에 입력하여 합성 360도 고해상도 영상인 t^SR을 생성한다.The arithmetic unit 530 is

The result of multiplying the mask M and the mask M for each element is input to the reconstruction layer 340 to generate a synthesized 360-degree high-resolution image t ^SR .

Meanwhile, the computing device 530 may synthesize a 360-degree image of the user's point of view to the user located at a specific point by using the 360-degree high-resolution image.

The output device 560 may output an interface screen of the super-resolution process. The output device 560 may output a high-resolution image that is a super-resolution result.

In addition, the super-resolution method as described above may be implemented as a program (or application) including an executable algorithm that can be executed in a computer. The program may be provided by being stored in a temporary or non-transitory computer readable medium.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the various applications or programs described above are CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM) Alternatively, it may be provided while being stored in a non-transitory readable medium such as an EEPROM (Electrically EPROM) or a flash memory.

Temporarily readable media include: Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (Enhanced) SDRAM, ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM) and Direct Rambus RAM (Direct Rambus RAM, DRRAM) refers to a variety of RAM.

This embodiment and the drawings attached to this specification merely clearly show a part of the technical idea included in the above-described technology, and within the scope of the technical idea included in the specification and drawings of the above-described technology, those skilled in the art can easily It will be apparent that all inferred modified examples and specific embodiments are included in the scope of the above-described technology.

Claims (12)

receiving, by the image processing apparatus, a low-resolution target image and a high-resolution reference image from among a plurality of 360-degree images providing a multi-view;
outputting, by the image processing apparatus, disparity information of the target image and the reference image by inputting the target image and the reference image to a disparity estimation model;
aligning, by the image processing apparatus, the reference image based on the disparity information; and
and outputting, by the image processing apparatus, a high-resolution image of the target image by inputting the target image and the aligned reference image to a reconstruction layer composed of residual blocks,
The target image and the reference image are images modeled as a 360-degree image based on an ERP (Equirectangular Projection),
The disparity estimation model performs a convolution operation on each of the target image and the reference image as shown in the following equation to extract features obtained by leveling pixels clustered according to latitude. Way.

(s(i,j) is the convolution operation for the position (i,j), s is an output feature value, w is a kernel, K is a value that sets the size of the kernel,

, H is the height of the frame) According to claim 1,
The target image and the reference image are synthesized 360-degree images,
ERP-based multi-view 360-degree image further comprising the step of the image processing device inputting each of the target image and the reference image, which are the actual 360-degree images, to the transition layer, and converting the target image and the reference image, which are the synthetic 360-degree images, respectively super-resolution method. According to claim 1,
The high-resolution image is a synthetic 360-degree image,
The ERP-based multi-view 360-degree image super-resolution method further comprising the step of the image processing device converting the high-resolution image to the transition layer into a live-action 360-degree image. According to claim 1,
The disparity estimation model extracts feature values in a plurality of hierarchical structures for each of the target image and the reference image, and provides matching information between the features of the target image extracted from the same layer and the features of the reference image to the flow estimator. An ERP-based super-resolution method of multi-view 360-degree images that input and output the disparity information. delete According to claim 1,
generating a mask by inputting, by the image processing apparatus, the target image, the aligned reference image, and the disparity information into a mask generation model;
The image processing apparatus outputs the high-resolution image by inputting a value obtained by multiplying the aligned reference image by the mask and the aligned reference image to the reconstruction layer,
The mask is an ERP-based super-resolution method of multi-view 360-degree images that removes a characteristic that degrades the performance of generating the high-resolution image from the input reference image. an input device for receiving a low-resolution target image and a high-resolution reference image from among a plurality of 360-degree images providing a multi-view;
a storage device for storing a neural network model that super-resolution a 360-degree image in consideration of the distortion between the ERP (Equirectangular projection) images; and
The target image and the reference image are input to the neural network model to generate disparity information of the target image and the reference image, and the target image and the reference image arranged based on the disparity information are used to generate the target image. Including a computing device for generating a high-resolution image for the image,
The target image and the reference image are images modeled as 360-degree images based on ERP,
The neural network model includes a mask generation layer that generates a mask by inputting the target image, the aligned reference image, and the disparity information, and a reconstruction layer that outputs a high-resolution image of the target image using the aligned reference image do,
The operation unit outputs the high-resolution image by inputting a value obtained by multiplying the aligned reference image and the mask and the aligned reference image to the reconstruction layer,
The mask is an ERP-based image processing apparatus for super-resolution 360-degree images that removes a characteristic that deteriorates the performance of generating the high-resolution image from the input reference image. 8. The method of claim 7,
The target image and the reference image are synthesized 360-degree images,
The neural network model includes a transition layer of the input stage,
The computing device inputs a target image and a reference image, which are actual 360-degree images, respectively, to the transition layer of the input stage, and super-resolution an ERP-based 360-degree image that generates the target image and the reference image, which are synthetic 360-degree images. processing unit. 8. The method of claim 7,
The high-resolution image is a synthetic 360-degree image,
The neural network model includes a transition layer of the output stage,
The arithmetic unit is an ERP-based 360-degree image processing apparatus for super-resolution by inputting the high-resolution image to the transition layer of the output stage and converting it into a live-action 360-degree image. 8. The method of claim 7,
The neural network model includes a disparity estimation layer,
The computing device inputs the target image and the reference image to the disparity estimation layer to extract feature values in a plurality of hierarchical structures, and matching information between the features of the target image extracted from the same layer and the features of the reference image An image processing apparatus for super-resolution an ERP-based 360-degree image that generates the disparity information by estimating the flow in the . 8. The method of claim 7,
The neural network model includes a disparity estimation layer,
The disparity estimation layer performs a convolution operation as shown in the following equation on each of the target image and the reference image to extract a feature obtained by leveling pixels clustered according to latitude. processing unit.

(s(i,j) is the convolution operation for the position (i,j), s is an output feature value, w is a kernel, K is a value that sets the size of the kernel,

, H is the height of the frame) delete

Images