KR102597298B1 - Image warping network system using local texture estimator - Google Patents

Abstract

The present invention relates to an image warping network system using local texture estimation, which utilizes both Fourier characteristics and Jacobian matrices of spatially varying coordinate transformation to correct image distortion and achieve high resolution (SR, Super- It is about technology that provides image data with resolution.

Description

Image warping network system using local texture estimator}

The present invention relates to an image warping network system using local texture estimation. More specifically, it utilizes both Fourier characteristics and Jacobian matrices of spatially varying coordinate transformations to correct image distortion and achieve high resolution ( This relates to an image warping network system using local texture estimation that can provide SR (Super-Resolution) image data.

Image warping is a geometric processing technique that moves the position of pixels that is widely used in various computer vision and graphics tasks, for example, image editing, optical flow, image alignment, and omnidirectional (omnidirectional) vision, etc.

At this time, while geometric processing such as enlarging or reducing the image can obtain uniform return results by applying certain rules to all pixels, image warping can vary the degree of movement for each pixel and can correct the distorted image. It is used to restore or convert to normal form.

Accordingly, depending on the viewpoint, image warping technology can be viewed as a generalized form of SR (Super-Resolution) technology, and based on the input data shown in a) of Figure 1, SR technology is shown in b) of Figure 1. , While the scale factor is expanded by applying the same rules to all areas, the image warping technology converts each location to a different scale factor, as shown in c) of Figure 1.

A common image warping technique is to find spatial locations in the input space and apply an interpolation kernel to compute missing RGB values. This interpolation-based image warping technique has the advantage of being comprised of a framework that is differentiable and easy to implement. On the other hand, there is a problem of low accuracy/reliability due to the presence of blurred areas in the resulting image (output image).

Recently, 'SRWarp' technology has been proposed to solve these problems, and SRWarp includes a homography transform using an adaptive warping layer, providing super-resolution to the existing image warping technology. By applying technology, we attempted to increase the resolution of the resulting image. However, there was a problem with poor performance for untrained transformations.

Accordingly, to solve this problem, another conventional technology recently applied Implicit Neural Representation to super-resolution technology to maintain high performance even for unlearned transformations, but this also failed to capture high-frequency region information, resulting in blurry output images. A visible problem appeared.

To solve this problem of not being able to capture high-frequency region information, another conventional technology, LTE (Local Texture Estimator), was recently proposed. However, this also has the problem of not being able to do image warping.

The reason why image warping technology is more difficult to implement than high-resolution technology is that although it performs high-resolution restoration for all positions, the scale factor is different for each position, so it has good performance even for unlearned transformations and cannot capture high-frequency region information. Technology for image warping that can be used is in demand.

In this regard, "CVPR 2022, Local Texture Estimator for Implicit Representation Function" proposes the LTE technology described above.

CVPR 2022, Local Texture Estimator for Implicit Representation Function

The present invention was created to solve the problems of the prior art as described above. The purpose of the present invention is to correct image distortion by utilizing both the Fourier characteristic and the Jacobian matrices of spatially varying coordinate transformation. and provides an image warping network system using local texture estimation that can provide high-resolution (SR, Super-Resolution) image data.

An image warping network system using local texture estimation according to an embodiment of the present invention, which includes an encoder module that receives basic image data and calculates the included feature map, pre-entered local grid information, and shape ( Shape) information and the feature map are input, estimate Fourier information considering coordinate transformation, and output a resample image using the estimated Fourier information and Jacobian matrices resulting from coordinate transformation. an image processing module, a decoder module that receives the resampled image and outputs estimated RGB information of the resampled image, a bilinear interpolation module that calculates bilinear interpolation image data of the basic image data, and the bilinear interpolation image It is desirable to include a summing module that adds data and estimated RGB information of the resampled image to provide a warped up-scaled image.

Furthermore, the encoder module is preferably composed of a deep SR (Super-Resolution) network that does not include an upscaling module, and the feature map is calculated through the deep SR network.

Furthermore, the deep SR network is preferably one selected from Enhanced Deep Super-Resolution Network (EDSR), Residual Channel Attention Network (RCAN), and Residual-in-Residual Dense Block network (RRDB).

Furthermore, the image processing module includes an amplitude estimator that estimates the amplitude using the feature map, a frequency estimator that estimates the frequency using the feature map, and a phase of the shape information. It further includes a phase estimator that estimates , and preferably performs a vector inner product using the local grid information and the estimated frequency.

Furthermore, it is preferable that the amplitude estimator and the frequency estimator are each formed as a 3×3 convolutional layer with 256 channels, and the phase estimator is formed as a single linear layer with 128 channels.

The image warping network system using local texture estimation of the present invention with the above configuration utilizes both Fourier characteristics and Jacobian matrices of spatially varying coordinate transformation to correct image distortion and achieve high resolution (SR). , Super-Resolution) has the advantage of being able to provide image data (warped upscaling image data).

Figure 1 is a configuration diagram showing an image warping network system using local texture estimation according to an embodiment of the present invention.
Figure 2 is an example diagram showing image warping data as a neural network by an image warping network system using local texture estimation according to an embodiment of the present invention.
Figure 3 is an example diagram showing a Jacobian matrix and a Hessian tensor applied to an image warping network system using local texture estimation according to an embodiment of the present invention.
Figure 4 is an exemplary diagram showing the learning process of an image warping network system using local texture estimation according to an embodiment of the present invention.
5 to 10 are comparison diagrams of output images of an image warping network system using local texture estimation according to an embodiment of the present invention and another image warping network system.
Figure 11 is an example diagram visualizing Fourier information applied to an image warping network system using local texture estimation according to an embodiment of the present invention.

Hereinafter, an image warping network system using local texture estimation of the present invention will be described in detail with reference to the attached drawings. The drawings introduced below are provided as examples so that the idea of the present invention can be sufficiently conveyed to those skilled in the art. Accordingly, the present invention is not limited to the drawings presented below and may be embodied in other forms. Additionally, like reference numerals refer to like elements throughout the specification.

At this time, if there is no other definition in the technical and scientific terms used, they have the meaning commonly understood by those skilled in the art to which this invention pertains, and the gist of the present invention is summarized in the following description and attached drawings. Descriptions of known functions and configurations that may be unnecessarily obscure are omitted.

In addition, a system refers to a set of components including devices, mechanisms, and means that are organized and interact regularly to perform necessary functions.

As described above, image warping is a technique for reconstructing an image defined in a rectangular grid into an arbitrary shape. Implicit neural representations (INR) are being applied to continuously express these images.

In existing computer vision research, when dealing with images, it is common to express them as a matrix with RGB values for each pixel position, and through this, the capacity required for storage and the model that can handle the image are generally determined according to the resolution of the image. Size depends. Recently, we want to express an image as a function that converts the x, y coordinates of the pixel location into r, g, and b values, and learning this function by expressing it with a neural network is a core technology of implicit neural expression. When an image is expressed as a function through implicit neural expression, the image can be stored regardless of resolution, can be applied to a computer vision model, and also has the advantage of being able to simply express SR technology that converts the image to high definition.

For example, each pixel in an image has a specific RGB value, and these pixels are combined to create an image. In order to express such an image as a function, coordinates (x, y) must be input and the value (RGB value or intensity value) of each pixel must be output, and this function expression itself corresponds to image data.

The Multi-Layer Perceptron (MLP) is designed to parameterize this INR and receive coordinates.

However, as mentioned in the background technology, the stand-alone multi-layer perceptron (MLP), which forms the artificial intelligence network for this function expression (implicit neural representation), has difficulty learning the high-frequency region (high-frequency Fourier coefficients). There is.

In other words, the disadvantage of implicit neural representation is that the independent multi-layer perceptron with ReLU learns mainly in the low-frequency region.

This image warping is used in a variety of computer vision and graphics tasks, such as image editing, optical flow, image alignment, and omnidirectional vision.

Conventionally, image warping was performed by applying inverse coordinate transformation to interpolate missing RGB values in the input space. However, this interpolation image warping technique results in blurred areas in the output image.

To solve this problem, the recent SRWarp technique adopts the Deep Single Image Super Resolution (SISR) architecture as its backbone and is designed to reconstruct images with high-frequency details.

The technical purpose of SISR is to reconstruct degraded low-resolution image data into high-resolution image data. Deep feature maps are extracted and upscaled at the end of the network to reconstruct them into high-resolution image data. Due to the nature of this technology, it is possible to output visually clear high-resolution image data, but because each local area of the distorted image is expanded by a different scale factor, another limitation exists.

In order to solve the problems of the prior art described above, the image warping network system using local texture estimation according to an embodiment of the present invention includes an encoder module, an image processing module, a decoder module, and a bilinear, as shown in FIG. It is composed of an interpolation module and a summation module, and each component is provided and operated in one operation processing means including a computer or in each operation processing means.

The image warping network system using local texture estimation according to an embodiment of the present invention considers both the Fourier information of the input image and the spatial change characteristics of the coordinate transformation, and formulates and expresses the frequency response estimator for image warping through INR. I do it.

Briefly about the image warping network system using local texture estimation according to an embodiment of the present invention, the input image ( ) and differentiable and reversible coordinate transformations ( ), as shown in Figure 2, I ^WARP ( ) relates to an algorithm for a local texture estimator (aka, 'LTEW, Local Texture Estimator for image Warping') for INR.

In geometry, the determinant of the Jacobian represents the local scale, so that the Fourier information is multiplied by the spatially varying Jacobian power for each pixel before the MLP represents I ^WARP .

An image warping network system using local texture estimation according to an embodiment of the present invention estimates Fourier information using an input image (basic image data) and utilizes the Jacobian matrix resulting from the estimated Fourier information and coordinate transformation. It is desirable.

Accordingly, the image warping network system using local texture estimation according to an embodiment of the present invention utilizes both Fourier information and the Jacobian matrix of spatially varying coordinate transformation to provide continuous neural representation for image warping. Algorithms can be provided. In other words, by using both the feature map of the deep neural network encoder and the relative coordinates or local grid information, it is possible to provide generalization for areas where learning has not occurred.

Before a detailed explanation, let's learn more about INR. Motivated by the fact that neural networks are general-purpose function approximators, INR is widely applied to represent continuous domain signals. Typically, memory requirements for data are quadratically or cubically proportional to signal resolution. In contrast, INR provides an efficient memory framework to store continuous signals because the storage size is proportional to the number of model parameters rather than signal resolution.

In addition, in order to solve the problem of bias in the frequency domain, the image warping network system using local texture estimation according to an embodiment of the present invention uses a local texture estimator (LTE) to generate low-resolution images synchronized by Fourier analysis. Fourier features for high-resolution image data are estimated from the data.

As shown in Figure 1, the input coordinate space is a set ), and the output coordinate space is Y( ) means.

In Local Neural Representation, neural representation ( ) is parameterized by an MLP with trainable parameters (θ). Decoding function ( ) is the query point ( ) is predicted through Equation 1 below.

here, and

and

Is is a set of

is the local ensemble coefficient,

means the latent variable for index j,

Is means the coordinates of

Through recent research, it has been found that the standard MLP structure has problems with learning in the high frequency region, and to solve this spectrum bias problem, Equation 1 above was modified as shown in Equation 2 below.

here, is a local texture estimator (LTE).

LTE( ) contains at least two estimators, one being an amplitude estimator ( ), and the other is a frequency estimator ( )am.

At this time, the estimation function ( ) is defined as Equation 3 below.

here, ,

,

is the local grid,

is the amplitude vector,

means the frequency matrix for exponent j,

<, > means vector inner product,

⊙ means element-wise multiplication.

Through the above-mentioned equations, Fj is the frequency response of the input image (I ^IN ), and since the frequency response is different from the warping image by I ^WARP , a high-resolution warping image cannot be output.

)을 아핀 변환(affine transformations)으로 선형화한다.To solve this problem, an image warping network system using local texture estimation according to an embodiment of the present invention provides a given coordinate transformation (

) is linearized with affine transformations.

For this, a local grid near the point (x _j ) ( ) The linear approximation of ) is calculated by Equation 4 below.

)의 야코비안 행렬이며, here, is a coordinate transformation (

) is the Jacobian matrix of

means the order condition of x ² or more,

refers to the local grid of the input space

By the affine theorem and Equation 4 above, the point ( ) Frequency response of the nearby warped image (I ^WARP ) ( ) is approximated by Equation 5 below.

In this way, through Equations 4 and 5 above, the estimation function of Equation 3 above is generalized as Equation 6 below.

Comparing Equation 3 and Equation 6 above, the LTEW expression is instead of input coordinate space It can be seen that Fourier information for the warped image can be extracted by utilizing the local grid of .

For SISR within a symmetric scale factor, the pixel shape of the upsampled image is square and spatially invariant.

However, in the case of image warping, pixels of the resampled image may have arbitrary shapes and vary spatially, as shown in a) of FIG. 3.

In order to solve this problem, in the present invention, the pixel shape ( ).

Here, [·,·] means connection after flattening,

, represents a numerical Jacobian matrix indicating the direction of the pixel and a numerical Hessian tensor specifying the degree of curvature, respectively.

To express shape information, the eight points closest to the query point ( ) is applied to the inverse coordinate transformation, the difference is calculated, and the numerical differentiation as shown in b) of FIG. 3 is calculated.

)이 클래스 C²에 있다고 가정하고, 이는 를 의미한다.Given coordinate transformation (

) is in class C ² , which gives means.

Accordingly, for shape expression Only six variables are used.

Shape information includes information about edge positions or pixel shapes, and accordingly, Equation 6 above is redefined as Equation 8 below.

here, is a phase estimator.

In addition, in the present invention, in order to stabilize network convergence and help LTEW learn high-frequency region information, bilinear interpolation image data ( ) will be used more.

Accordingly, the local neural representation of the warped image (I ^WARP ) using LTEW proposed by the present invention is defined as Equation 9 below.

To this end, let us look in detail at each configuration of the image warping network system using local texture estimation according to an embodiment of the present invention.

The encoder module ( ) It is desirable to receive basic image data and calculate a feature map (z) included in the basic image data.

At this time, the encoder module ( ) is preferably composed of a deep SR network that does not include an upscale module, and a feature map is calculated using the constructed deep SR network.

The deep SR network is preferably one selected from conventional technologies such as EDSR (Enhanced Deep Super-Resolution Network), RCAN (Residual Channel Attention Network), and RRDB (Residual-in-Residual Dense Block network), and is limited to this. It's not like that.

The image processing module ( ) is pre-entered local grid information ( ), shape information (s), and characteristic map (z) by the encoder module are input, and Fourier information considering coordinate transformation is estimated. In addition, a resample image is output using the estimated Fourier information and Jacobian matrices resulting from coordinate transformation.

), 주파수 추정기(

) 및 위상 추정기(

)를 포함하게 된다.At this time, the image processing module ( ) is an amplitude estimator (

), frequency estimator (

) and phase estimator (

) will be included.

)는 상기 특성 맵을 이용하여, 진폭(amplitude)을 추정하며, 상기 주파수 추정기(

)는 상기 특성 맵을 이용하여, 주파수(frequency)를 추정하게 된다.The amplitude estimator (

) uses the characteristic map to estimate the amplitude, and the frequency estimator (

) uses the characteristic map to estimate the frequency.

)는 상기 형상 정보의 위상(phase)을 추정하는 것으로, 상기의 수학식 8과 같이, 상기 로컬 그리드 정보()와 추정한 주파수를 이용하여 벡터 내적(inner product)을 수행하는 것이 바람직하다.In addition, the phase estimator (

) is to estimate the phase of the shape information, and as in Equation 8 above, the local grid information ( It is desirable to perform a vector inner product using ) and the estimated frequency.

)와 주파수 추정기(

)는 각각 256 채널을 갖는 3×3 컨볼루션 레이어로 형성되는 것이 바람직하며, 상기 위상 추정기(

)는 128 채널을 갖는 단일 선형 레이어로 형성되는 것이 바람직하다.At this time, the amplitude estimator (

) and frequency estimator (

) is preferably formed of 3×3 convolutional layers each having 256 channels, and the phase estimator (

) is preferably formed as a single linear layer with 128 channels.

) 근처에서 동일한 텍스처를 갖는다고 가정하고, 상기 진폭 추정기(

)와 주파수 추정기(

)는 각각 가장 가까운 보간(Nearest-neighborhood interpolation) 기법을 이용하여,

에서 푸리에 정보(

)를 추정하는 것이 바람직하다.In the present invention, the warping image is the point (

), assuming that they have the same texture nearby, the amplitude estimator (

) and frequency estimator (

) each uses the nearest-neighborhood interpolation technique,

Fourier information (

) is desirable to estimate.

In addition, the image processing module ( ) is preferably multiplied by the amplitude and the sine wave activation output before generating the resample image.

The decoder module ( ) is the image processing module ( ) is input, and estimated RGB information of the resample image is output.

At this time, the decoder module ( ) is formed as a four-layer MLP with ReLU and contains 256 hidden dimensions.

The bilinear interpolation module calculates bilinear interpolation image data of the basic data.

The summation module sums the bilinear interpolation image data and the estimated RGB information of the resample image and provides (outputs) a warped up-scaled image.

To explain the learning process of the imaging warping network system using local texture estimation including the above-described configurations,

Assume two groups of size B, one group is the image group ( ), and another group is the coordinate transformation group ( ), where each is differentiable and reversible.

Prepare input image data to which inverse coordinate transformation is applied as shown in Equation 10 below.

here, am.

Afterwards, cropped image data is generated by cropping the input image data, avoiding void pixels. ( , here, lim.)

Meanwhile, the ith group element ( , here ), randomly sample the query points (M) among the valid coordinates and prepare the correct answer data (GT, Ground Truth).

Afterwards, as shown in FIG. 4, in order to evaluate the warped up-scaled image for each query point according to the present invention, the loss is calculated by comparing it with the correct data group (see Equation 11 below).

here, means RGB information.

In the case of a high-resolution technique with asymmetric scale (asymmetric-scale SR), each scale factor (sx, sy) is randomly sampled from u(0.25, 1), and in the case of homography transform, the equation below Inverse coordinate transformation is randomly sampled from the distribution by 12.

Here, hx, hy ~ u(0.25, 0.25) means sheering,

means rotation,

sx, sy ~ u(0.35, 0.5) means scaling,

tx to u(-0.75W, 0.125W), ty to u(0.75H, 0.125H), px to u(-0.6W, 0.6W), py to u(-0.6H, 0.6H) are projections. ) means.

As a result of evaluating the generalization performance of the present invention through unseen transformation, in the case of high-resolution techniques with asymmetric scale and homography transformation, sx, sy ~ u (0.125, 0.25) for unlearned coordinate transformation Sampling was performed, and it was found that other parameters (hx, hy, θ, tx, ty, px, py) were the same.

Furthermore, various experiments were performed to evaluate the performance of the present invention.

In detail, for the high-resolution technique with asymmetric scale, Peak Signal-to-Noise Ratio (PSNR) for network evaluation was analyzed for Set5, Set14, B100, and Urban100 (see Table 1 below), and as input data 48 × 48 patches were used, and bicubic resizing was performed in Pytorch for random scale down sampling during the learning process. Additionally, it was designed as an encoder without an upscaling module.

In the case of homography transformation, for network evaluation, homography transformation was applied to the existing high-resolution images of Set5, Set14, B100, and Urban100, and the Set5-warping, Set14-warping, B100-warping, and Urban100-warping data sets were analyzed. During the learning process, bicubic resampling was used for random homography transformation, the maximum size of the input match was 48 × 48 patches, and an encoder without an upscaling module was designed.

In addition, the results of the image warping network system using local texture estimation according to an embodiment of the present invention are learned to represent homography transformation, and ERP perspective projection is performed to evaluate generalization performance without additional learning.

As a result of the evaluation, the image warping network system using local texture estimation according to an embodiment of the present invention, the conventional RCAN, MetsSR (Magnification Arbitrary Network for Super-Resolution), and ArbSR (Scale- The results of comparison with Arbitrary Super-Resolution are shown in Table 1 and Figure 5 (in-scale) below and Table 2 and Figure 6 (out-of-scale) below.

Here, red means the best performance, and blue means the next performance.

For RCAN, low-resolution images are upsampled by 4 times and resampled using bicubic interpolation. In the case of MetaSR, the input image is upsampled by the maximum (sx, sy) multiple and downsampled using the bicubic technique.

Except for Set14 and B100, it can be seen that the image warping network system using local texture estimation according to an embodiment of the present invention is superior to the performance of the conventional network system in terms of performance and visual quality for all scale factors and all data sets. there is.

In addition, the results of comparison between the image warping network system using local texture estimation according to an embodiment of the present invention by applying homography transformation and the conventional RRDB and SRWarp are shown in Table 3, Figures 7, and 8 below.

Here, red means the best performance, and blue means the next performance.

In the case of RRDB, the input image is supersampled 4 times and converted using the bicubic resampling technique.

It can be seen that the image warping network system using local texture estimation according to an embodiment of the present invention is superior to the performance of the conventional network system in mPSNR and visual quality for both in-scale and out-of-scale.

In addition, the verification results of generalization ability applying ERP perspective projection are shown in Figure 9. In the case of RRDB, the input ERP image was upsampled 4 times and interpolated using the bicubic technique. At this time, the resolution of the input ERP image was 1644 × 832.

However, considering limited hardware resources such as HMD, it is impossible to save and transmit the ERP image at full resolution, so the high-resolution ERP image is downsampled by 4 times and the image is converted to 832 Project at size 832.

As shown in FIG. 9, RRDB clearly has limitations in showing high-frequency areas, and SRWarp shows structures near the boundary, but in the image warping network system using local texture estimation according to an embodiment of the present invention, It can be seen that the resulting data shows fine details without structures near the boundary.

In addition, in Figure 10, as a result of projecting a high-resolution ERP image with a size of 8190 × 2529 to 4095 × 4095 with a FOV of 90°, RRDB and SRWarp were unable to analyze it.

In other words, both RRDB and SRWarp need to upsample the input ERP image by 4 times, which is practically impossible because a huge amount of memory is required.

On the other hand, since the image warping network system using local texture estimation according to an embodiment of the present invention can query analysis points sequentially, it can be seen that restoration is possible by efficiently utilizing memory.

In addition, in Figure 11, the Discrete Fourier Transform (DFT) feature space is visualized to verify the image warped by the image warping network system using local texture estimation according to an embodiment of the present invention.

In detail, the estimated frequency is scattered in 2D space and the color is set according to the size. After this, the input image ( Before generating a resample image using the Fourier information (Fj) estimated from ), the Fourier space (Fj) is the Jacobian matrix ( ), the output image ( ) is consistent with the frequency response of

As described above, the present invention has been described with reference to specific details such as specific components and drawings of limited embodiments, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above-mentioned embodiment. No, those skilled in the art can make various modifications and variations from this description.

Accordingly, the spirit of the present invention should not be limited to the described embodiments, and all matters that are equivalent or equivalent to the claims of this patent as well as the claims described below shall fall within the scope of the spirit of the present invention. .

Claims (5)

An encoder module that receives basic image data and calculates an included feature map;
Receives pre-entered local grid information, shape information, and the feature map, estimates Fourier information considering coordinate transformation, and Jacobian matrices resulting from the estimated Fourier information and coordinate transformation. An image processing module that outputs a resample image using;
a decoder module that receives the resampled image and outputs estimated RGB information of the resampled image;
a bilinear interpolation module that calculates bilinear interpolation image data of the basic image data; and
a summation module that adds the bilinear interpolated image data and estimated RGB information of the resampled image to provide a warped up-scaled image;
Image warping network system using local texture estimation, including.
According to clause 1,
The encoder module is
An image warping network system using local texture estimation, which consists of a deep SR (Super-Resolution) network that does not include an upscaling module, and calculates a feature map through the deep SR network.
According to clause 2,
The deep SR network is
An image warping network system using local texture estimation, which is one of EDSR (Enhanced Deep Super-Resolution Network), RCAN (Residual Channel Attention Network), and RRDB (Residual-in-Residual Dense Block network).
According to clause 2,
The image processing module is
an amplitude estimator that estimates amplitude using the characteristic map;
a frequency estimator that estimates frequency using the characteristic map; and
a phase estimator that estimates the phase of the shape information;
It further includes,
An image warping network system using local texture estimation that performs a vector inner product using the local grid information and the estimated frequency.
According to clause 4,
The amplitude estimator and the frequency estimator are
It is formed of 3×3 convolutional layers with 256 channels each,
The phase estimator is
Image warping network system using local texture estimation, formed from a single linear layer with 128 channels.