RU2652722C1 - Data processing for super-resolution - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: invention relates to the field of digital image processing and video. Super-resolution processing unit includes an offset estimator configured to receive a set of low resolution images of one scene, receive pixel offsets for low resolution images for all pixels corresponding to the same fragments in the low resolution image set, receive sets of integer pixel offsets by calculating an integer pixel offset for each pixel offset, receive sets of fractional pixel offsets by calculating a fractional pixel offset for each pixel offset; a filter bank configured to store filter sets; filter selection block; a high-resolution image acquisition unit configured to obtain a high resolution image in an RGB format.

EFFECT: improve image and video quality without losing image data.

18 cl, 5 dwg, 9 tbl

Description

BACKGROUND OF THE INVENTION

FIELD OF THE INVENTION

[0001] The present invention relates to the field of digital image and video processing, in particular to an apparatus and method for processing super resolution data, and is intended to improve the quality of images and video by improving resolution, which is also called super resolution of images and video.

DESCRIPTION OF THE PRIOR ART

[0002] Traditional technologies for improving image and video quality have the following disadvantages. Iterative approaches are excessively slow and have high complexity due to high computational costs. Non-iterative approaches do not take into account fractional displacements of pixels, which leads to a decrease in the quality of the resulting images. In pipelined approaches, image data is lost and the directivity of the texture of small parts may deteriorate. There is no compromise between noise reduction (NR) and super resolution (SR).

[0003] There are various technologies for improving the quality of images and videos. One technology is described in the publication of patent application US 20130064474 A1, published March 14, 2013 and entitled "Calculation of images with higher resolution from multiple images with lower resolution." The superresolution problem with a formulation using a posteriori maximum is solved using the nonlinear iterative optimization method using the conjugate gradient method. However, the use of iterations leads to high computational costs.

[0004] Other technologies are described in US patent US 9282253 B2, published on 03/08/2016 and entitled "System and method for interpolation of super-resolution based on multiple frames for digital cameras." A digital camera system is provided, which includes a motion registration module generating motion information, an interpolation module, a weighting coefficient calculation module, and a weight combining module. The system supports Bayer input. Each input frame is interpolated independently of the other frames. However, subpixel (fractional) displacement between frames is not used when restoring parts.

[0005] Since traditional technologies for improving the quality of images and videos have the above-mentioned disadvantages, an approach for improving the quality of images and videos is needed, which provides accurate reproduction of the directivity of the texture, low computational costs and a good compromise between noise reduction and super resolution. The present invention is intended to eliminate at least the problems and disadvantages described above, and to provide at least the advantages described below.

SUMMARY OF THE INVENTION

[0006] It is an object of the present invention to provide an apparatus and method for processing super-resolution data that improves image and video quality without losing image and video data and provides low computational costs.

[0007] According to one aspect of the invention, there is provided an apparatus for processing superresolution data, comprising: an offset estimation unit configured to: receive a set of low-resolution images of one scene, obtain sets of pixel offsets for low-resolution images for all pixels corresponding to the same fragments in a set of low-resolution images by calculating pixel offsets for pixels corresponding to the same fragments in low-resolution images, relative Only the pixels corresponding to the same fragments in the first low-resolution image selected from the set of low-resolution images, obtain sets of integer pixel offsets by calculating the integer pixel offsets for each pixel offset, while the sets of integer pixel offsets correspond to the sets of pixel offsets for low-resolution images, obtain sets of fractional pixel offsets by calculating fractional pixel offsets for each pixel offset, wherein the set fractional pixel displacements correspond to displacements of sets of pixels for the low-resolution images; a filter bank configured to store filter sets, the filter sets being pre-computed; a filter selection unit configured to select filter sets from a filter bank based on sets of fractional pixel offsets; a high-resolution image acquisition unit configured to obtain a high-resolution image in RGB format by computing pixels of a high-resolution image using a set of low-resolution images, selected filter sets, and sets of integer pixel offsets.

[0008] In a further aspect, pre-computed filter sets are obtained by: setting a finite number of fractional pixel offsets; generating sets of fractional pixel offsets by selecting all combinations of fractional pixel offsets; computing filter sets based on sets of fractional pixel shifts and a regularizer describing local constraints for superresolution.

[0009] In another further aspect, the number of filters stored in the filter bank is reduced based on at least one of the conditions for matching pixel offsets.

[0010] In yet a further aspect, filter sets stored in the filter bank include omnidirectional filter sets and directional filter sets, wherein undirected filters are associated with image areas containing an undirected texture, and directional filters are associated with image areas containing an undirected texture .

[0011] In yet a further aspect, the apparatus further comprises a texture direction estimation unit configured to determine texture directions of low resolution images, the filter selection unit further configured to select either sets of omnidirectional filters or sets of directional filters based on sets of fractional pixel offsets and texture directions in low resolution images.

[0012] In yet a further aspect, the device further comprises: a preprocessing unit configured to: determine noise reduction parameters for all pixels in low resolution images including a noise model, image fragment size, search radius for adjacent image fragments, and standard deviation dependence pixel noise from pixel brightness; calculating texture-dependent noise reduction coefficients based on changes in pixel brightness, wherein each texture-dependent noise reduction coefficient is calculated for each pixel of low resolution images; estimating the standard deviations of the pixel noise using the dependence of the standard deviation of the pixel noise on the pixel brightness, texture-dependent noise reduction coefficients and the noise reduction coefficient specified by the user; performing three-dimensional noise reduction (3DNR) on a set of low-resolution images based on noise reduction parameters and estimated pixel noise standard deviation values to obtain low-resolution images with reduced noise; and a validity map estimation unit configured to generate a validity map using a set of low-resolution images and sets of pixel offsets, the validity map representing the reliability of calculating pixel offsets, and the texture direction estimating unit is further configured to determine texture directions of low-resolution images with reduced noise; wherein the filter selection unit is further configured to select either sets of non-directional filters or sets of directional filters based on a confidence map, sets of fractional pixel offsets and texture directions in low-resolution images with reduced noise; wherein the high-resolution image acquisition unit is further configured to receive the high-resolution image in RGB format by calculating the pixels of the high-resolution image using a confidence map, a set of low-resolution images with reduced noise, selected filter sets and sets of integer pixel offsets.

[0013] In yet a further aspect, the low resolution images of one scene are RGB images.

[0014] In yet a further aspect, the low resolution images of one scene are Bayer images.

[0015] In yet a further aspect, the device further comprises a post-processing unit configured to: convert the high-resolution image in RGB format obtained by the high-resolution image acquisition module into an YUV image; process the image in YUV format by: calculating a map of standard deviations of pixels in channel Y using a sliding window method, calculating a reference channel Ref by smoothing a map of standard deviations of pixels using Gaussian filtering, cross-channel filtering of channels U and V using the reference channel Ref; and reverse converting the processed YUV image to the post-processed high-resolution RGB image.

[0016] According to another aspect of the invention, there is provided a method for processing data for super-resolution, comprising the steps of: receiving a set of low-resolution images of one scene; get sets of pixel offsets for low-resolution images for all pixels corresponding to the same fragments in the set of low-resolution images by calculating pixel offsets for pixels corresponding to the same fragments in low-resolution images relative to pixels corresponding to the same fragments in the first a low resolution image selected from a set of low resolution images; obtaining sets of integer pixel offsets by calculating integer pixel offsets for each pixel offset, wherein the sets of integer pixel offsets correspond to sets of pixel offsets for low resolution images; obtaining sets of fractional pixel offsets by calculating fractional pixel offsets for each pixel offset, wherein the sets of fractional pixel offsets correspond to sets of pixel offsets for low resolution images; selecting filter sets from pre-calculated filter sets based on sets of fractional pixel offsets; obtaining a high-resolution image in RGB format by computing pixels of a high-resolution image using a set of low-resolution images, selected filter sets, and sets of integer pixel offsets.

[0017] In one additional aspect, pre-computed filter sets are obtained by: setting a finite number of fractional pixel offsets; generating sets of fractional pixel offsets by selecting all combinations of fractional pixel offsets; computing filter sets based on sets of fractional pixel shifts and a regularizer describing local constraints for superresolution.

[0018] In another further aspect, the number of said filters is reduced based on at least one of the conditions for matching pixel offsets.

[0019] In yet a further aspect, pre-computed filter sets include omnidirectional filter sets and directional filter sets, wherein undirected filters are associated with image areas containing an undirected texture, and directional filters are associated with image areas containing an undirected texture.

[0020] In another further aspect, the method further comprises determining the texture directions of the low resolution images, the selection of filter sets comprising selecting either sets of non-directional filters or sets of directional filters based on sets of fractional pixel offsets and texture directions in low images permissions.

[0021] In yet a further aspect, the method further comprises the steps of: determining noise reduction parameters for all pixels in low resolution images, including a noise model, image fragment size, search radius for neighboring image fragments, and the dependence of the standard deviation of pixel noise from pixel brightness calculating texture dependent noise reduction coefficients based on changes in pixel brightness, wherein each texture dependent noise reduction coefficient is calculated for each pixel of low resolution images; evaluating the standard deviation of the pixel noise using the dependence of the standard deviation of the pixel noise on the pixel brightness, texture-dependent noise reduction coefficients and the noise reduction coefficient specified by the user; performing three-dimensional noise reduction (3DNR) on a set of low-resolution images based on noise reduction parameters and estimated pixel noise standard deviation values to obtain low-resolution images with reduced noise; and generating a confidence map using a set of low-resolution images and sets of pixel offsets, wherein the confidence map represents the reliability of calculating pixel offsets, while determining the directions of the texture of the low-resolution images includes determining the directions of the texture of the low-resolution images with reduced noise; wherein the selection of filter sets comprises a selection of either sets of non-directional filters or sets of directional filters based on a confidence map, sets of fractional pixel offsets and texture directions in low-resolution images with reduced noise; however, obtaining a high-resolution image in RGB format includes computing pixels of a high-resolution image using a confidence map, a set of low-resolution images with reduced noise, selected filter sets and sets of integer pixel offsets.

[0022] In yet a further aspect, the low resolution images of one scene are RGB images.

[0023] In yet a further aspect, the low resolution images of one scene are Bayer images.

[0024] In another further aspect, the method further comprises the steps of: converting a high-resolution image in RGB format to an image in YUV format; process the image in YUV format by: calculating a map of standard deviations of pixels in channel Y by a sliding window method, calculating a reference channel Ref by smoothing a map of standard deviations of pixels using Gaussian filtering, cross-bilateral filtering of channels U and V using the reference channel Ref; and perform the inverse conversion of the processed image in YUV format to the post-processed high-resolution image in RGB format.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other aspects, features and advantages of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings, in which:

[0026] FIG. 1 is a block diagram of one embodiment of a super-resolution data processing apparatus.

[0027] FIG. 2 is a block diagram of another embodiment of a super-resolution data processing apparatus.

[0028] FIG. 3 is a block diagram of yet another embodiment of a super-resolution data processing apparatus.

[0029] FIG. 4 illustrates vector reduction with components to the first or second quadrant when estimating the direction of the image texture.

[0030] FIG. 5 illustrates the separation of pixels of low-resolution Bayer images into separate color channels G1, G2, R, and B.

[0031] In the following description, unless otherwise indicated, the same reference numerals are used for the same elements when depicted in different drawings, and a description thereof will be omitted.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

[0032] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of various embodiments of the present disclosure, as defined by the claims and their equivalents. It includes various specific details to help in this understanding, but they should only be considered as exemplary. Accordingly, those skilled in the art will understand that various changes and modifications to the various embodiments described herein can be made without departing from the scope of the present disclosure. In addition, descriptions of well-known functions and constructions may be omitted for clarity and conciseness.

[0033] The terms and words used in the following description and claims are not limited to bibliographic meanings, but are only used by the inventor to provide a clear and consistent understanding of the present disclosure. Accordingly, it should be apparent to those skilled in the art that the following description of the various embodiments of the present disclosure is for illustration only.

[0034] It should be understood that the terms in the singular do not exclude plurality, unless expressly indicated otherwise.

[0035] It should be understood that, although the terms first, second, etc., may be used here in relation to the elements of the present disclosure, such elements should not be construed as limited to these terms. Terms are used only to indicate the difference of one element from other elements.

[0036] In addition, it should be understood that the terms “comprises”, “comprising”, “includes” and / or “including” when used herein indicate the presence of the indicated features, values, operations, elements and / or components but do not exclude the presence or addition of one or more other features, values, operations, elements, components and / or groups thereof.

[0037] In various embodiments of the present disclosure, a “module” or “block” may perform at least one function or operation and may be implemented using hardware, software, or a combination thereof. A “plurality of modules” or “a plurality of blocks” can be implemented by at least one processor (not shown) by integrating it with at least one module or block other than a “module” or “block” that must be implemented using a particular hardware.

[0038] Various embodiments of the present disclosure are described in more detail below with reference to the accompanying drawings.

[0039] The present invention can obtain a high-resolution image from both a set of low-resolution images of one scene in RGB format (additive color model) and a set of Bayer (raw format) low-resolution images of one scene received from an image sensor. The resulting high-resolution image is in RGB format. Unlike the prior art, the processing in accordance with the present invention is performed in an iterative manner with filtering based on sets of fractional pixel offsets. In addition, embodiments of the present invention take into account the directivity of the texture of images by using estimates of the directivity of the texture and filtering using directional and non-directional filters. Additionally, in embodiments of the present invention, three-dimensional noise reduction is used in combination with super-resolution operations to achieve a good compromise between noise reduction and super-resolution, which are used individually in the prior art.

Super Resolution Basic Principle

[0040] Typically, a super-resolution task is reduced to finding one high-resolution image from several low-resolution images. At the same time, when restoring pixels, high-resolution images use pixels from the original low-resolution images, taking into account fractional (sub-pixel) pixel offsets.

известно, тогда изображения низкого разрешения

,

, ….

можно найти, произведя трансформацию входного изображения и последующую понижающую дискретизацию. В качестве трансформации может быть проективное или аффинное преобразование, а также любое непрерывное преобразование координат. Трансформация обычно обусловлена движением камеры между моментами съемки входных изображений низкого разрешения, также она частично может быть обусловлена смещением объектов внутри кадра. При съемке изображений низкого разрешения с использованием реальной оптической системы, помимо трансформации и понижающей дискретизации, применяемым к идеальному изображению высокого разрешения, применяется еще и сглаживание, обусловленное функцией рассеяния точки (PSF) оптической системы. Формирование изображений низкого разрешения из изображения высокого разрешения происходит в соответствии с формулой:

,[0041] The essence of the method is as follows. Let the original high resolution image

known then low resolution images

,

, ...

can be found by transforming the input image and subsequent downsampling. As a transformation, there can be a projective or affine transformation, as well as any continuous transformation of coordinates. The transformation is usually due to the movement of the camera between the moments of shooting input images of low resolution, and it can also be partially due to the displacement of objects within the frame. When shooting low-resolution images using a real optical system, in addition to the transformation and downsampling applied to an ideal high-resolution image, smoothing is also used due to the point spread function (PSF) of the optical system. The formation of low-resolution images from high-resolution images occurs in accordance with the formula:

,

- оператор понижающей дискретизации,

- оператор сглаживания,

- оператор трансформации для получения k-го изображения, а

- шум. Оператор

можно оценить экспериментально, теоретически на основе детального описания оптической системы, или подобрать приблизительно исходя из критериев качества результирующего изображения. Оператор трансформации находят с использованием методов оценки смещения. При этом подразумевается, что

и

- это векторизованные изображения низкого разрешения. Если исходное изображение в двумерном виде было представлено матрицей

,Where

- downsampling operator,

- smoothing operator,

is the transformation operator to obtain the kth image, and

- noise. Operator

can be estimated experimentally, theoretically based on a detailed description of the optical system, or selected approximately based on the quality criteria of the resulting image. The transformation operator is found using bias estimation methods. It is understood that

and

- These are low resolution vectorized images. If the original image in two-dimensional form was represented by a matrix

,

that vectorized image can be represented as

, сжимающая изображение размера

в

раз по вертикали и в

раз по горизонтали будет иметь вид

, где

- Кронекерово произведение,

- единичная матрица размера

, а

-

-й столбец матрицы

Downsampling Matrix

compressing size image

at

times vertically and in

times horizontally will have the form

where

- Kronecker work,

- unit size matrix

, but

-

matrix column

, исходную матрицу

в векторизованной форме надо домножить на матрицуSo, in order to reduce the above matrix by 2 times and get the matrix

, the original matrix

in vectorized form you need to multiply by the matrix

and convert from a vector of length 4 to a 2 × 2 matrix.

и изображения низкого разрешения размером

матрицы

и

будут иметь размер

, а матрица

- размер

. В дальнейшем без потери общности ограничимся рассмотрением квадратных изображений размером

.[0042] Thus, for high resolution image size

and low resolution image size

matrices

and

will be sized

, and the matrix

- the size

. In the future, without loss of generality, we restrict ourselves to considering square images of size

.

[0043] For a given low-resolution image formation model, the high-resolution image search task can be formulated as minimizing the functionality specified by the formula

.

.

[0044] This problem can be formulated using various norms, but the solution is easiest to find in the case of a quadratic norm, which we will continue to use.

тождественный (т.е. нет сглаживания), а оператор трансформации представляет собой циклическое смещение на

пикселей по вертикали и

пикселей по горизонтали. Тогда матрицу этого оператора можно построить как

, где

матрица оператора, осуществляющего циклическое смещение столбцов матрицы размером

на

.[0045] Sometimes this problem can be solved directly. In one embodiment of the invention, the smoothing operator

identical (i.e., there is no smoothing), and the transformation operator is a cyclic shift by

pixels vertically and

pixels horizontally. Then the matrix of this operator can be constructed as

where

matrix of the operator performing cyclic shift of matrix columns of size

on

.

[0046] In order to perform an offset in the matrix

на 1 пиксель по вертикали и 2 пикселя по горизонтали и получить матрицу

можно построить матрицу трансформации

1 pixel vertically and 2 pixels horizontally and get the matrix

can build a transformation matrix

полученных из изображения высокого разрешения при помощи смещений

и понижающей дискретизации в 2 раза. Тогда очевидно, что исходное изображение можно просто «собрать» из пикселей изображений низкого разрешения.[0047] Let us have 4 low resolution images

obtained from a high resolution image using offsets

and 2 times downsampling. Then it is obvious that the original image can simply be “assembled” from pixels of low-resolution images.

, метод выдавал разумное решение (т.е. при входных изображениях с одинаковым смещением происходила пространственная интерполяция с усреднением), на решение накладываются дополнительные ограничения. Эти ограничения формулируются в виде регуляризирующего члена

(в дальнейшем - «регуляризатор»), прибавляемого к минимизируемому функционалу, заданному формулой:[0048] If, for example, offsets in some images were repeated, then a complete reconstruction would become impossible. In order that in cases when a complete reconstruction is impossible, or the task is ill-conditioned (what happens, including due to the presence of a smoothing operator

, the method produced a reasonable solution (i.e., spatial interpolation with averaging occurred for input images with the same offset), additional restrictions are imposed on the solution. These restrictions are formulated as a regularizing term.

(hereinafter referred to as the “regularizer”), added to the minimized functional defined by the formula:

[0049] Usually formulated in this way, the superresolution problem is solved using iterative methods that do not allow real-time solutions. To solve the problem in real time, approximate methods are used.

Solution of the Super Resolution Problem According to the Present Invention

[0050] In the present invention, we obtain an exact solution to the problem in real time without using iterations.

, где

- скалярный параметр регуляризации, а

- некий оператор. Тогда оптимизируемый функционал можно записать в виде формулы:[0051] We use the Tikhonov regularization, ie we solve the optimization problem for the regularizer

where

is the scalar parameter of regularization, and

- a certain operator. Then the optimized functional can be written in the form of a formula:

Where

[0052] As is known, an optimization problem of this kind has an analytical solution expressed by the formula:

можно найти в виде линейной комбинации пикселей входных изображений низкого разрешения по формуле:

[0053] This means that each pixel of a high resolution image with coordinates

can be found as a linear combination of pixels of input low-resolution images by the formula:

- пиксель

-го входного изображения низкого разрешения с координатами

, а

- целое число, соответствующее коэффициенту понижающей дискретизации (во сколько раз размер изображения высокого разрешения отличается от размера изображения низкого разрешения). Значение весового коэффициента

можно найти из вычисленной матрицы

, как ее элемент с координатами

:Where

- pixel

low-resolution input image with coordinates

, but

- an integer corresponding to the downsampling coefficient (how many times the size of the high resolution image differs from the size of the low resolution image). Weight coefficient

can be found from the calculated matrix

as its element with coordinates

:

where parentheses indicate the capture of the corresponding matrix element.

, которые соответствуют одному и тому же смещению всех пикселей изображения высокого разрешения, во-вторых, рассматриваем только такие операторы

, которые представляют собой свертку (или циклическую свертку) с некоторым ненулевым ядром

с суммой элементов, равной нулю, например,

.[0054] Since it is impossible to save (such an amount of space is required that is currently not available for use) of solving all possible problems for all possible offsets, we introduced two additional restrictions: first, we consider only such transformations

that correspond to the same offset of all pixels of a high resolution image, secondly, we consider only such operators

which are convolution (or cyclic convolution) with some nonzero kernel

with the sum of elements equal to zero, for example,

.

задавать не

матрицами размера размером

, а

парами смещений (по вертикали и горизонтали):[0055] The first limitation allows decision parameters

ask not

size matrices

, but

pairs of displacements (vertical and horizontal):

для всевозможных сочетаний смещений

.[0056] Reconstruction of the high resolution image is based on the preservation of pre-calculated and stored matrices

for all sorts of combinations of offsets

.

достаточно хранить лишь по нескольку элементов из каждой матрицы

при этом нет необходимости проводить вычисления для всех возможных смещений, достаточно сохранить результаты лишь для некоторых наборов

таких, что

. Более того, в результате построения такой аппроксимации отпадает необходимость исходного предположения, что смещения пикселей в изображениях низкого разрешения друг относительно друга (и относительно изображения высокого разрешения) одинаковы для всех пикселей изображения. Для корректного функционирования настоящего изобретения достаточно условия, чтобы лишь соседние пиксели изображения низкого разрешения в окрестности размером

имели одинаковые смещения относительно соответствующих пикселей в других изображениях низкого разрешения.[0057] According to the present invention, to provide a very accurate approximation

it is enough to store only a few elements from each matrix

there is no need to carry out calculations for all possible displacements; it is enough to save the results only for some sets

such that

. Moreover, the construction of such an approximation eliminates the need for the initial assumption that the pixel offsets in low-resolution images relative to each other (and relative to the high-resolution image) are the same for all pixels in the image. For the present invention to function correctly, it is sufficient that only neighboring pixels of a low resolution image in a neighborhood of size

had the same offset relative to the corresponding pixels in other low-resolution images.

Description of the invention

[0058] FIG. 1 shows a block diagram of a super-resolution data processing apparatus 100. The device 100 comprises an offset estimation unit 101, a filter selection unit 102, a filter bank 103, a high resolution image acquisition unit 104. The super-resolution data processing apparatus 100 can obtain a high resolution RGB image from both a set of low resolution images of one scene in RGB format and a set of low resolution Bayer images of one scene

пикселей выходного изображения с координатами

, причем

- коэффициент понижающей дискретизации, выполняют следующие операции:[0059] The offset estimation unit 101 receives a set of low-resolution images of one scene and obtains sets of pixel offsets for low-resolution images for all pixels. We accept as a condition that the first image has a zero offset, and offsets for other images will be calculated relative to the first image. If a set of low-resolution images of one scene contains images in RGB format, then for each

pixels of the output image with coordinates

, and

- downsampling coefficient, perform the following operations:

,- calculate the coordinates of the corresponding pixel in the first low-resolution image

,

относительно пикселя с координатами

в первом изображении низкого разрешения пикселей других изображений низкого разрешения, соответствующих тому же фрагменту для изображения.- calculate offsets

relative to the pixel with coordinates

in the first low-resolution image, pixels of other low-resolution images corresponding to the same fragment for the image.

вычисляют целочисленное смещение

и дробное смещение

, где квадратные скобки соответствуют операции взятия целой части.[0060] Then, the offset estimation unit 101 obtains sets of integer pixel offsets by calculating integer pixel offsets for each pixel offset, wherein the sets of integer pixel offsets correspond to sets of pixel offsets for low resolution images, and obtains sets of fractional pixel offsets by calculating fractional pixel offsets for each pixel offset, with sets of fractional pixel offsets corresponding to sets of pixel offsets for low resolution images. For each of the pixel offsets of low-resolution images

calculate integer offset

and fractional displacement

where the square brackets correspond to the operation of taking the integer part.

для некоторого набора смещений

. При этом благодаря первому и второму ограничениям, описанным в разделе «Решение задачи сверхразрешения согласно настоящему изобретению» для восстановления каждого пикселя выходного изображения требуется не больше

пикселей входного изображения. Банк 103 фильтров содержит совокупность наборов фильтров, вычисленных для всевозможных сочетаний

таких, что

, где

- количество используемых изображений низкого разрешения.[0061] Filter bank 103 stores pre-computed filter sets. A set of filters is a set of values by which the matrix can be uniquely restored.

for some set of offsets

. Moreover, due to the first and second limitations described in the section “Solving the Super Resolution Problem According to the Present Invention” no more is needed to restore each pixel of the output image

pixels of the input image. Bank 103 filters contains a set of filter sets calculated for all kinds of combinations

such that

where

- the number of low resolution images used.

выбирают соответствующий набор фильтров

где каждый из фильтров

состоит из

величин (весовых коэффициентов фильтра):[0062] The filter selection unit 102 is for selecting filter sets from a filter bank based on sets of fractional pixel offsets. For each fractional offset for low resolution images

select the appropriate filter set

where each of the filters

consists of

values (filter weights):

и

,

and

,

- весовые коэффициенты фильтра.Where

- filter weights.

, такие, что[0063] The high-resolution image acquisition unit 104 obtains a high-resolution image in RGB format by computing pixels of a high-resolution image using a set of low-resolution images, selected filter sets, and sets of integer pixel offsets. Obtaining high-resolution images in RGB format is as follows. First, pixel sets are selected from the low-resolution images.

such that

,

,

- пиксель из входного изображения с номером

с координатами

. Затем вычисляют каждый из пикселей блока выходного изображения с координатами

как сумму скалярных произведений

и

для

:Where

- pixel from input image with number

with coordinates

. Then, each of the pixels of the output image block with the coordinates is calculated

as the sum of scalar products

and

for

:

[0064] The above implementation allows calculating a high resolution image in real time in any digital camera with a burst function.

обрабатывают группы не

, а по

пикселей, где 3 - это количество выходных цветовых каналов.[0065] The difference between processing a set of low-resolution Bayer images and low-resolution image processing in RGB format is only that with a downsampling ratio

do not handle groups

, and

pixels, where 3 is the number of output color channels.

между изображениями

и

в точке изображения с координатами

.[0066] The calculation of fractional pixel offsets in Bayer images is also different from the calculation of fractional pixel offsets in RGB images. For Bayer images, the classic Lucas-Canada optical flux method is used. The method is as follows: let it be required to find the offset

between images

and

at the image point with coordinates

.

и вертикальному и горизонтальному направлению -

и

, соответственно.For this, time gradients are calculated

both vertical and horizontal direction -

and

, respectively.

, гдеThe bias is found by solving the equation

where

.

.

также зависит от

и

, поиск этих величин является итерационным процессом. В качестве нулевого приближения используется нулевое смещение. Для получения более надежных результатов при построении

и

вместо отдельных значений градиентов берется их сумма в пределах

- окрестности точки

Insofar as

also depends

and

The search for these quantities is an iterative process. Zero bias is used as the zeroth approximation. For more reliable results when building

and

instead of individual values of gradients, their sum is taken within

- neighborhood of a point

In order for the method to behave reasonably in low-textured areas of the image, it makes sense to use regularization

As another improvement, you can use anti-aliasing with a Gaussian window

и

для трехканальных изображений производится сложение градиентов по всем трем каналам. Для вычисления градиентов для изображений в Байеровском формате мы использовали формулы:Since the Lucas-Canada method is designed to determine displacements with an amplitude of up to one pixel, to estimate the displacements in real images that can reach several pixels, several calculation methods can be used, for example, as a non-limiting example, a multiscale scheme can be used without anisotropic filtering. In addition, as a non-limiting example, you can use 3 levels - ¼ resolution in YCbCr format, ½ resolution in YCbCr format and the full image in Bayer format. When building matrices

and

for three-channel images, gradients are added over all three channels. To calculate the gradients for images in Bayer format, we used the formulas:

получено путем билинейной интерполяции Байеровского изображения с использованием пикселей одного и того же цветаwhere is the meaning

obtained by bilinear interpolation of a Bayer image using pixels of the same color

где

- операция округления вниз. Для зеленых пикселей билинейная интерполяция применялась к пикселям

,

где

- операция округления к ближайшему целому.for blue and red,

Where

- rounding operation down. For green pixels, bilinear interpolation was applied to pixels

,

Where

- rounding operation to the nearest integer.

где

- это номер цветового, соответствующий набору смещений

, где каждый из фильтров

состоит из

величин (весовых коэффициентов фильтра):[0067] The operation of selecting a filter set for Bayer images is also different from selecting a filter set for images in RGB format. For Bayer images in the filter selection block 102, a filter set is selected from the filter bank 103

Where

is the color number corresponding to the set of offsets

where each of the filters

consists of

values (filter weights):

и

,

and

,

- весовые коэффициенты фильтра.Where

- filter weights.

для каждого цветового канала

как сумму скалярных произведений

и

для

:[0068] The calculation of pixels of a high resolution image using a set of low resolution Bayer images is different from the above calculation of pixels of a high resolution image using a set of low resolution RGB images in that each of the pixels of the output image block with coordinates

for each color channel

as the sum of scalar products

and

for

:

[0069] The preliminary calculation of the filters for the bank of 103 filters is performed on a high-performance computer according to the following procedure:

;- choose the size of the resulting image in high resolution image, for example,

;

для изображения размером

;- construct a matrix of the regularization operator

for image size

;

, где

- степень двойки, то для получения изображения высокого разрешения из любого набора смещений, вычисленного с доступной точностью, строят наборы фильтров для конечного числа наборов смещений

, где

и каждое из остальных смещений может принимать одно из

значений- if the existing method for determining fractional displacement has accuracy

where

is a power of two, then to obtain a high-resolution image from any set of offsets calculated with available accuracy, filter sets are constructed for a finite number of sets of offsets

where

and each of the other displacements can take one of

values

эта совокупность наборов смещений является базовым набором;

this set of displacement sets is a basic set;

из базового набора выполняют следующие операции:- for each set of offsets

from the base set perform the following operations:

вычисляют оператор трансформации

, производящий соответствующее смещение пикселей изображения размером

,- for each pair of offsets

compute the transformation operator

producing an appropriate pixel offset of the image size

,

,- build a matrix

,

,- build a matrix

,

для

при помощи следующих операций:- select from the matrix the elements corresponding to the filter elements select a set of filters

for

using the following operations:

в изображении размером

так, чтобы

и

были кратны коэффициенту понижающей дискретизации

,- choose a central pixel with coordinates

in image size

so that

and

were multiples of the downsampling coefficient

,

выбирают из матрицы

строку под номером

и обозначают

,- for everybody

choose from matrix

line number

and denote

,

выделяют

последовательно идущих подстрок

длины

каждая,- from each line

emit

consecutive substrings

lengths

each

преобразуют в матрицу

так, чтобы элемент вектора с номером

становился элементом матрицы с индексами:- each of the substrings

convert to matrix

so that the vector element with number

became an element of the matrix with indices:

- операция округления внизWhere

- rounding down operation

выбирают

коэффициентов

фильтра

так, что- from each matrix

choose

coefficients

filter

so that

.

.

[0070] The preliminary calculation of the filters for the bank of 103 filters for a set of Bayer low-resolution images differs from the corresponding solution for images in the RGB format by the following operations:

, то рассматривается базовый набор смещений, где каждое смещение принимает одно из возможных

значений от

до

,- if the existing method for determining fractional displacements of pixels has accuracy

, then we consider a basic set of offsets, where each offset takes one of the possible

values from

before

,

,- build a matrix

,

- оператор понижающей дискретизации для изображений в формате RGB,

- оператор сглаживания для изображений в формате RGB,

- оператор трансформации для получения k-го изображения в формате RGB. При этом операторы для изображений в формате RGB получают из соответствующих операторов для одноканальных изображений согласно формулам:Where

- downsampling operator for images in RGB format,

- smoothing operator for images in RGB format,

- transformation operator for obtaining the k-th image in RGB format. Moreover, operators for images in RGB format are obtained from the corresponding operators for single-channel images according to the formulas:

можно построить в виде:Bayer downsampling operator

can be built in the form:

,

,

- оператор понижающей дискретизации в 2 раза по вертикали и горизонтали, соответственно, а

- оператор циклического сдвига на

столбцов по горизонтали и

строк по вертикали.Where

- the operator of downsampling 2 times vertically and horizontally, respectively, and

- cyclic shift operator by

horizontal columns and

rows vertically.

,- build a matrix

,

для изображений в формате RGB строят в виде:Cross channel regularizer

for images in RGB format build in the form:

,

,

Where

выбирают из матрицы

три строки с номерами

,

и

, соответствующие трем цветовым каналам выходного изображения и обозначают

,- for everybody

choose from matrix

three lines with numbers

,

and

corresponding to the three color channels of the output image and denote

,

делят на

равных последовательно идущих подстрок

,- from each line

divided by

equal consecutive substrings

,

делят на 4 равные последовательные части

и преобразуют каждую из этих частей в матрицу

и

, соответственно (размер каждой из матриц

),

так, чтобы элемент вектора с номером

превращался в элемент матрицы с индексами:- each of the substrings

divided into 4 equal consecutive parts

and transform each of these parts into a matrix

and

, respectively (the size of each of the matrices

),

so that the vector element with number

turned into a matrix element with indices:

- операция округления вниз,Where

- rounding operation down

выбирают по

весовых коэффициентов

фильтра

размером

так, что:- of four matrices

choose by

weighting factors

filter

the size

so that:

уровней квантования смещения,

входных изображений и коэффициенте увеличения в

раз потребуется

наборов по

фильтров. Эксперименты показали, что для получения изображения высокого разрешения из изображений в формате RGB достаточно фильтров размером 11×11. Поскольку при использовании операций сверх разрешения для восстановления тонких деталей имеют ценность только дробные смещения пикселей, для надежного их восстановления необходимо, чтобы смещения между пикселями входных изображений было невелико. При пакетной съемке с использованием, например, камеры мобильного телефона без применения штатива этого можно достигнуть только при небольшом количестве входных изображений (не более 4-х). Точность надежной оценки дробных смещений пикселей не превышает

пикселя. Количество величин, которые необходимо вычислить и сохранить для некоторых применимых на практике сочетаний точности оценки смещения, количества входных изображений и коэффициента увеличения приведено в таблице B01.[0071] For

bias quantization levels,

input images and magnification in

times required

sets of

filters. The experiments showed that 11 × 11 filters are sufficient to obtain high-resolution images from RGB images. Since when using operations beyond the resolution to restore thin details, only fractional pixel shifts are of value, to reliably restore them, it is necessary that the shifts between the pixels of the input images are small. In batch shooting using, for example, a mobile phone camera without using a tripod, this can be achieved only with a small number of input images (no more than 4). Accuracy of reliable estimation of fractional pixel displacements does not exceed

pixels. The number of values that need to be calculated and stored for some practical combinations of the accuracy of the bias estimate, the number of input images and the magnification factor is given in table B01.

Table B01

Number of input images

Zoom ratio

Pixel Offset Estimation Accuracy

Number of Filter Sets The number of filters in the set The total number of stored values, Mb 2 2 1/2 four 8 0.003692627 3 2 1/2 16 12 0.02 four 2 1/2 64 16 0.12 2 3 1/4 16 eighteen 0.03 3 3 1/4 256 27 0.80 four 3 1/4 4096 36 02.17 2 four 1/4 16 32 0.06 3 four 1/4 256 48 1.42 four four 1/4 4096 64 30.25

уровней квантования смещения,

входных изображений и коэффициенте увеличения в

раз потребуется

наборов по

фильтров (множитель 3 возникает по причине наличия 3х цветовых каналов изображения - R (красного), G (зеленого) и B (синего)). Эксперименты показали, что для получения изображения высокого разрешения из байеровских изображений достаточно фильтров размером 16×16, а при небольшой потере точности достаточно и 12×12. Количество величин, которые необходимо вычислить и сохранить для получения изображения высокого разрешения из байеровских изображений для некоторых применимых на практике сочетаний точности оценки смещения пикселей, количества входных изображений и коэффициента увеличения приведено в таблице B02.[0072] Similarly, to obtain a high resolution image from input Bayer images for

bias quantization levels,

input images and magnification in

times required

sets of

filters (multiplier 3 occurs due to the presence of 3 color channels of the image - R (red), G (green) and B (blue)). The experiments showed that to obtain a high-resolution image from Bayer images, 16 × 16 filters are enough, and with a small loss of accuracy, 12 × 12 are enough. The number of values that must be calculated and stored in order to obtain a high-resolution image from Bayer images for some practical combinations of the accuracy of estimating the pixel displacement, the number of input images, and the magnification factor are given in Table B02.

Table B02

Number of input images

Zoom ratio

Pixel Offset Estimation Accuracy

Number of Filter Sets The number of filters in the set The total number of stored values, Mb 2 2 1/2 16 96 0.38 3 2 1/2 256 144 9.00 four 2 1/2 4096 192 192.00 2 3 1/4 64 216 3.38 3 3 1/4 4096 324 324.00 four 3 1/4 262144 432 27648.00 2 four 1/4 64 384 6.00 3 four 1/4 4096 576 576.00 four four 1/4 262144 768 49152.00

[0073] From the above tables it is seen that the number of stored values is extremely large. To reduce this amount, i.e. to reduce the number of filters stored in the filter bank 103, the present invention proposes to use the following principles as the conditions for matching pixel offsets.

[0074] The condition for the coincidence of pixel offsets (1) is that if two sets of offsets coincide up to a change in the order of input images, then the calculated filters will coincide up to an order of images.

и

смещения совпадают с точностью до замены координат

, то и вычисленные фильтры будут совпадать с точностью до порядка координатных осей.[0075] Condition for matching pixel offsets (2) - if in two sets of offsets

and

displacements coincide up to a change of coordinates

, then the calculated filters will coincide up to the order of the coordinate axes.

и

смещения совпадают с точностью до целочисленных смещений

и

:

, то вычисленные фильтры будут совпадать, а при применении фильтров пиксели из изображений низкого разрешения следует брать с учетом смещений

и

.[0076] Condition for matching pixel offsets (3) - if in two sets of offsets

and

displacements coincide up to integer displacements

and

:

, then the calculated filters will coincide, and when applying filters, pixels from low-resolution images should be taken taking into account the offsets

and

.

и

, смещения совпадают с точностью до знака

, где

, то вычисленные фильтры будут совпадать с точностью до зеркального отображения относительно осей абсцисс и ординат, соответственно.[0077] Condition for matching pixel offsets (4) - if in two sets of offsets

and

, displacements coincide up to a sign

where

, then the calculated filters will coincide up to a mirror image relative to the abscissa and ordinate, respectively.

, то в наборе фильтров можно хранить значения, соответствующие только одному из этих изображений.[0078] A condition for matching pixel offsets (5) if some offsets coincide in the set of offsets, for example,

, then in the filter set you can store values that correspond to only one of these images.

). Количество фильтров в исходном банке фильтров оценивается как

. При

рассматриваются следующие наборы смещений (предполагая

):[0079] An example shows how these principles can be applied to compress a filter bank in the 1st row of a table (RGB image,

) The number of filters in the original filter bank is estimated as

. At

the following sets of offsets are considered (assuming

):

Table B03

Set No.

It is reduced to a set using property (1) It is reduced to a set using property (2) It is reduced to a set using the properties (1) and (2) Shortcut entry one 0 0 0 0 one one one + 2 0 0 0

5 3 9 + 3 0 0

0 9 2 5 - four 0 0

13 four 13 + 5 0

0 0 2 9 3 - 6 0

0

6 eleven eleven + 7 0

0 10 10 7 + 8 0

fourteen 12 fifteen + 9

0 0 0 3 5 2 - 10

0 0

7 7 10 - eleven

0

0 eleven 6 6 - 12

0

fifteen 8 fourteen - 13

0 0 four 13 four - fourteen

0

8 fifteen 12 - fifteen

0 12 fourteen 8 - 16

16 16 16 +

и т.п. В 6-8 столбцах таблицы перечислены наборы, которые могут быть получены из текущего набора благодаря свойствам (1) и (2). В 9-й графе выбраны по порядку те наборы, которые необходимы для порождения всех прочих. В рассматриваемом случае достаточно рассмотреть 7 из 16 наборов. В настоящем примере свойство (4) не было применено, однако, если рассмотреть

, то применение свойства (4) позволит сократить количество рассматриваемых смещений с 256 до 34.[0080] Due to property (3), we may not include the quantities

etc. The 6-8 columns of the table list the sets that can be obtained from the current set due to properties (1) and (2). In the 9th column, those sets are selected in order that are necessary for generating all the others. In this case, it suffices to consider 7 of the 16 sets. In the present example, property (4) was not applied, however, if we consider

, then the use of property (4) will reduce the number of displacements under consideration from 256 to 34.

раз для порождения каждого блока выходного изображения размером

фильтры для получения каждой точки можно параметризовать дробным смещением между интерполируемой точкой и ближайшими точками из изображений низкого разрешения. Каждый пиксель из блока можно охарактеризовать дробным смещением относительно его верхнего левого угла в координатах низкого разрешения. Например, для

пиксели из блока

соответствуют смещениям

,

,

;

,

,

[0081] When enlarging an image in

times to spawn each output image block of size

Filters for obtaining each point can be parameterized with a fractional shift between the interpolated point and the nearest points from low-resolution images. Each pixel from the block can be characterized by a fractional shift relative to its upper left corner in low-resolution coordinates. For example, for

pixels from block

correspond to offsets

,

,

;

,

,

где

- номер текущего набора среди перенумерованных всевозможных смещений, тогда каждый фильтр, соответствующий этому набору, можно характеризовать смещением

, где

, а

и

пробегают всевозможные значения от 0 до

. Тогда фильтры, соответствующие некоторым смещениям

и

(т.е. могут рассматриваться фильтры, построенные для разных наборов смещений и для разных дробных смещений относительно верхнего левого угла блока), будут удовлетворять свойствам, аналогичным свойствам (1)-(5).[0082] Suppose there is a set of offsets

Where

- the number of the current set among the renumbered all kinds of offsets, then each filter corresponding to this set can be characterized by an offset

where

, but

and

run through all sorts of values from 0 to

. Then the filters corresponding to some offsets

and

(i.e., filters constructed for different sets of displacements and for different fractional displacements relative to the upper left corner of the block can be considered) will satisfy properties similar to properties (1) - (5).

и

совпадают с точностью до целочисленных сдвигов, то вычисленные фильтры будут совпадать, а при применении фильтров пиксели из изображений низкого разрешения следует брать с учетом соответствующих сдвигов.[0083] Condition for matching pixel offsets (6) - if offsets

and

coincide up to integer shifts, then the calculated filters will coincide, and when applying filters, pixels from low-resolution images should be taken taking into account the corresponding shifts.

и

совпадают с точностью до перемены порядка входных изображений, то и вычисленные фильтры будут совпадать с точностью до порядка изображений.[0084] The condition for matching pixel offsets (7) is if the offsets

and

coincide up to a change in the order of input images, then the calculated filters will coincide up to an order of images.

и

совпадают с точностью до замены координат

, то и вычисленные фильтры будут совпадать с точностью до порядка координатных осей.[0085] Condition for matching pixel offsets (8) - if offsets

and

coincide up to a change of coordinates

, then the calculated filters will coincide up to the order of the coordinate axes.

и

совпадают с точностью до знака, то вычисленные фильтры будут совпадать с точностью до зеркального отображения относительно осей абсцисс и ординат, соответственно.[0086] Condition for matching pixel offsets (9) - if offsets

and

coincide up to a sign, then the calculated filters will coincide up to a mirror image relative to the abscissa and ordinate, respectively.

некоторые пары элементов совпадают, например,

, то в наборе фильтров можно хранить значения, соответствующие только одному из изображений с номерами

и

.[0087] The condition for matching pixel offsets (10) if the offset

some pairs of elements match, for example,

, then in the filter set you can store values corresponding to only one of the images with numbers

and

.

.[0088] We show the application of these properties in the same example. For 7 sets of displacements included in the reduced set (Table B03), we compose the displacement vectors for s

.

Table B04

Unique Set No.

,

,

,

,

,

one

2

3

four

5

6

7

[0089] Using property (6), we transform the offsets from Table B04 to the range 0 ... 1.

Table B05

Unique Set No.

,

,

,

,

,

one

2

3

four

5

6

7

[0090] For the convenience of further analysis, group the elements from Table B05 according to the number of zeros.

Table B06

No. p / p Unique Set No. No. of pixel offsets during interpolation Pixel offsets relative to the interpolated point Number of zeros Possible repetitions one one 0

6 one 2 one one

3 2,3,11,18,19,21,24,26,27 3 one 2

3 2,3,11,18,19,21,24,26,27 four one 3

0 four 5 2 0

5 5.15 6 2 one

2 6,12,16,20,22,25 7 2 2

four 7,9,10,13,17,23,28 8 2 3

one 8.14 9 3 0

four 7,9,10,13,17,23,28 10 3 one

four 7,9,10,13,17,23,28 eleven 3 2

3 2,3,11,18,19,21,24,26,27 12 3 3

2 6,12,16,20,22,25 13 four 0

four 7,9,10,13,17,23,28 fourteen four one

one 8.14 fifteen four 2

5 5.15 16 four 3

2 6,12,16,20,22,25 17 5 0

four 7,9,10,13,17,23,28 eighteen 5 one

3 2,3,11,18,19,21,24,26,27 19 5 2

3 2,3,11,18,19,21,24,26,27 twenty 5 3

2 6,12,16,20,22,25 21 6 0

3 2,3,11,18,19,21,24,26,27 22 6 one

2 6,12,16,20,22,25 23 6 2

four 7,9,10,13,17,23,28 24 6 3

3 2,3,11,18,19,21,24,26,27 25 7 0

2 6,12,16,20,22,25 26 7 one

3 2,3,11,18,19,21,24,26,27 27 7 2

3 2,3,11,18,19,21,24,26,27 28 7 3

four 7,9,10,13,17,23,28

[0091] Some of the elements listed in the 6th column of Table B07 can be reduced to one another using properties (6) and (7).

Table B07

Number of zeros Possible repetitions Pixel offsets relative to the interpolated point 0 four

one 8.14 2 6,12,16,20,22,25 3 2,3,11,18,19,21,24,26,27 four 7,9,10,13,17,23,28 5 5.15 6 one

[0092] Property (8) is completely similar to property (4), therefore, it can be used at high magnification factors. Displacements from which due to properties (6) - (8) all other offsets can be generated are given in Table B08. In total, we can limit ourselves to considering 13 biases. Using property (10), you can also store only a subset of filters corresponding to each bias. The number of filters for each offset is indicated in the 4th column of Table B08. Thus, to obtain all the pixels of a high-resolution image for any offset, it is enough to save 26 filters instead of 192 original ones.

Table B08

No. p / p No. of a payment order from Tab. B04 Pixel offsets Number of filters one one

one 2 2

one 3 four

one four 5

2 5 6

2 6 7

2 7 8

2 8 9

2 9 eleven

2 10 12

2 eleven 17

3 12 eighteen

3 13 twenty

3 Total 26

[0093] The result of applying the compression method for 3 input images is shown in table B09

Table B09

Input image format

Number of input images

Zoom ratio

Pixel Offset Estimation Accuracy

Number of filters in a full filter bank Number of filters in a compressed filter bank Compression ratio RGB 3 2 ½ 192 26 7.38 RGB 3 four ¼ 12288 300 40.96 Bayer pattern 3 2 ½ 36864 450 81.92 Bayer pattern 3 four ¼ 2359296 25752 91.62

[0094] FIG. 2 is a block diagram of another embodiment of a super-resolution data processing apparatus 100 that takes into account the directivity of the image texture when acquiring a high resolution image from a set of low resolution images of one scene.

[0095] In the embodiment of FIG. 2 of the embodiment, the super resolution data processing apparatus 100 further includes a texture direction estimating unit 105, which is designed to determine texture directions of low resolution images. In addition, in this embodiment, the filter sets stored in the filter bank include omnidirectional filter sets and directional filter sets. Non-directional filters are filters associated with areas of an image containing an undirected texture, and directional filters are filters associated with areas of an image containing an undirected texture.

[0096] The filter selection unit 102 is further configured to select either sets of omnidirectional filters or sets of directional filters based on sets of fractional pixel offsets and texture directions in low resolution images determined by the texture direction estimation unit 105.

[0097] The texture direction estimation unit 105 performs determination of the texture directions of the low-resolution images as follows.

[0098] Traditionally, a structural tensor is used to estimate the local direction. For this, a 2 × 2 symmetric matrix of the structural tensor is made up of the sums of squares of vertical and horizontal pixel brightness gradients (changes in pixel brightness) calculated in a certain neighborhood, as well as the sum of the products of vertical and horizontal pixel brightness gradients:

и собственные вектора

этой матрицы по формуле:[0099] Then calculate the eigenvalues

and eigenvectors

this matrix by the formula:

,

,

,

,

. На основании собственных чисел вычисляется когерентность

. Если когерентность не превышает предварительно заданного порога, это означает либо то, что рассматриваемая точка относится к слабо текстурированной области, либо к сильно текстурированной, но не имеющей одного наиболее выраженного направления. В противном случае вычисляется доминирующее направление текстуры в рассматриваемой точки, при этом, если наибольшим по модулю собственным числом является

то направление вычисляется по формуле:

, а в противном случае по формуле:

.Where

. Based on the eigenvalues, coherence is calculated

. If the coherence does not exceed a predetermined threshold, this means either that the point in question refers to a weakly textured region, or to a strongly textured, but not having one, most pronounced direction. Otherwise, the dominant direction of the texture at the point in question is calculated, if the largest eigenvalue is

then the direction is calculated by the formula:

, otherwise, by the formula:

.

[0100] When processing images in RGB format, gradients are calculated as:

, где

- пиксель изображения

с координатами

и

.

where

- image pixel

with coordinates

and

.

[0101] To implement the present invention, a special technique has been developed for calculating the structural tensor for Bayer images, in which the gradients are calculated according to the following formulas:

- at the points of a Bayer pattern of type r and b, the gradients in the green channel are calculated according to the formulas:

- at the points of a Bayer pattern of type r, the gradients in the blue channel are calculated according to the formulas:

- at the points of a Bayer pattern of type r, the gradients in the red channel are calculated according to the formulas:

- at the points of a Bayer pattern of type b, the gradients in the red channel are calculated according to the formulas:

- at the points of a Bayer pattern of type b, the gradients in the blue channel are calculated according to the formulas:

- at the points of the Bayer pattern type g1, the horizontal gradient of the red channel and the vertical gradient of the blue channel are calculated according to the formulas:

- at the points of the Bayer pattern type g2, the horizontal gradient of the blue channel and the vertical gradient of the red channel are calculated according to the formulas:

- at the points of the Bayer pattern type g1, the horizontal gradient of the blue channel and the vertical gradient of the red channel are calculated according to the formulas:

- at the points of the Bayer pattern type g2, the horizontal gradient of the red channel and the vertical gradient of the blue channel are calculated according to the formulas:

- at the points of the Bayer pattern type g1 and g2, the gradients in the green channel are calculated according to the formulas:

- for each point of the input image, the gradients are calculated according to the formulas:

и поиск ее собственных векторов и значений

производятся также, как это производится и для изображений в формате RGB, однако перед вычислением доминирующего направления, вектора приводятся к первому или второму квадранту способом, показанным на фиг. 4. После вычисления арктангенса эта величина квантуется на выбранное количество уровней (например, N=8, N=16 или N=32) и прибавляется единица (нулевой банк фильтров предусмотрен для ненаправленных текстур). Это число указывает номер банка фильтров, который следует использовать при выборе набора фильтров для каждой точки.[0102] the compilation of the matrix

and search for its own vectors and values

are produced in the same way as for RGB images, however, before calculating the dominant direction, the vectors are reduced to the first or second quadrant in the manner shown in FIG. 4. After calculating the arc tangent, this value is quantized to the selected number of levels (for example, N = 8, N = 16 or N = 32) and one is added (a zero filter bank is provided for non-directional textures). This number indicates the number of filter banks to use when selecting a filter set for each point.

[0103] Additional restrictions, such as noise present in low-resolution input images, inaccurate estimation of pixel offsets in some areas of low-resolution images, or objective impossibility to estimate pixel offsets, may be considered to obtain a high-resolution image as a result of processing super-resolution low-resolution images. due to a sharp change in images, for example, caused by a change in lighting, as well as the presence of residual color artifacts in the output a battle. Also, when obtaining a high-resolution image, parameters that are most suitable for the characteristics of a particular device that captured low-resolution images can be used.

[0104] An embodiment of the super-resolution data processing apparatus 100 shown in FIG. 3 allows you to solve the above problems. The super resolution data processing device 100 further comprises a preprocessing unit 106, which, taking into account the low-resolution image pickup parameters, provides adaptive noise reduction, in particular three-dimensional noise reduction (3DNR). In addition, the super resolution data processing apparatus 100 further comprises a validity map estimator 107 that generates a validity map representing the reliability of the calculation of pixel offsets. The confidence map reveals pixels for which the offset is likely to be incorrectly determined.

[0105] An ultra-resolution data processing apparatus 100 comprising a preprocessing unit 106 and a validity map estimator 107 may process both RGB images and Bayer images.

[0106] The offset estimation unit 101 does not use data obtained after 3DNR, since it is advisable to use images not subjected to 3DNR to obtain a more reliable result.

[0107] The filter bank 103 is modified to store filters calculated for k, k-1, k-2, ... 1 low-resolution images when reconstructing a high-resolution image from k input low-resolution images.

[0108] The filter selection block 102 is modified so that if the offset for the current pixel is not calculated reliably for all input low-resolution images, then filters are selected for reconstruction from low-resolution images based on fractional pixel offsets from those low-resolution images, for which the offset is calculated reliably.

[0109] The high-resolution image acquisition unit 104 applies the super-resolution method to low-resolution images that have been pre-processed taking into account image areas where pixel offsets are not correctly calculated. Areas of images where pixel offsets are not correctly calculated are indicated on the confidence map.

[0110] If the pixel offsets are calculated reliably for all k low-resolution input images, the high-resolution image obtaining unit 104 calculates the pixels of the high-resolution image using filters for k images and the pixels of the input images from k low-resolution images subjected to 3DNR.

[0111] If the pixel offsets are not calculated reliably for all k input low-resolution images, the high-resolution image acquisition unit 104 selects the input pixels from those low-resolution images for which integer pixel offsets are reliably calculated, calculates the pixels of the high-resolution image using filters for reconstruction low-resolution images obtained from block 102 of the filter set.

[0112] The preprocessing unit 106 determines noise reduction parameters for all pixels in low resolution images, including a noise model, image fragment size, search radius for adjacent image fragments, and the dependence of the standard deviation of pixel noise on pixel brightness; calculates texture-dependent noise reduction coefficients based on changes in pixel brightness, with each texture-dependent noise reduction coefficient calculated for each pixel of low-resolution images; estimates the standard deviation of the pixel noise using the dependence of the standard deviation of the pixel noise on the pixel brightness, texture-dependent noise reduction coefficients and the noise reduction coefficient specified by the user; and performs 3DNR on a set of low-resolution images based on noise reduction parameters and estimated pixel noise standard deviation values to obtain low-resolution images with reduced noise.

[0113] Texture-dependent noise reduction coefficients represent an estimate of the degree of texturing of the image and indicate areas that potentially contain high-frequency parts that can be reconstructed using superresolution operations, and for which, in order to provide better visual quality, 3DNR should be carried out to a lesser extent than for non-textured areas and areas having one dominant direction.

, используемой для вычисления весовых коэффициентов фильтра, которые используются для вычисления усредненных значений пикселей.[0114] 3DNR is determined by the noise model, which may be Gaussian or Poisson; the size of the image fragment r, with which the comparison of other image fragments with which the averaging is performed; the search radius of neighboring fragments R, which determines the maximum distance at which fragments of the image averaged with the current fragment can be located; estimated values of the standard deviation of pixel noise at a given point in the image

used to calculate filter weights that are used to calculate average pixel values.

, размер фрагмента изображения

, радиус поиска соседних фрагментов

и зависимость стандартного отклонения шума пикселей от яркости пикселей

определяются для всего изображения в зависимости от времени экспозиции и коэффициента усиления

при этом большему времени выдержки и большему коэффициенту усиления соответствует Гауссова модель и меньшие значения радиусов, а меньшему времени выдержки и меньшему коэффициенту усиления соответствует Пуассоновская модель и большие значения радиусов. При этом конкретный способ выбора определяется данными из калибровочных таблиц, данные которых определяются экспериментально, либо сообщаются производителем камеры.[0115] The noise model

, image fragment size

, the radius of the search for neighboring fragments

and dependence of the standard deviation of pixel noise on pixel brightness

determined for the whole image depending on exposure time and gain

in this case, the Gaussian model and smaller radii correspond to a longer exposure time and a larger gain, and the Poisson model and large radii correspond to a shorter exposure time and a smaller gain. In this case, the specific selection method is determined by the data from the calibration tables, the data of which are determined experimentally, or reported by the camera manufacturer.

[0116] Texture-dependent noise reduction coefficients are calculated using the following steps:

- for the input image, gamma correction of the brightness values of pixels of low-resolution images is performed;

матрицы структурного тензора Т, используемого при оценке направленности текстуры изображения;- for each pixel of the low-resolution input image, a second eigenvalue is calculated

matrix of the structural tensor T used in assessing the directivity of the image texture;

, в котором значения

, превышающие выбранный эмпирически первый порог, устанавливаются равными первому порогу, а значения, меньшие выбранного эмпирически второго порога, устанавливаются равными второму порогу),- perform threshold conversion

in which values

exceeding the empirically selected first threshold are set equal to the first threshold, and values smaller than the empirically selected second threshold are set equal to the second threshold),

для получения значений

в диапазоне от 0 до 1, такая операция известна в уровне техники и ее подробное описание не требуется,- perform normalization of the converted values

to get the values

in the range from 0 to 1, such an operation is known in the prior art and its detailed description is not required,

с помощью фильтрации низкочастотным фильтром,- produce smoothing of normalized values

by filtering with a low-pass filter,

, причем инвертированные значения

являются текстурно-зависимыми коэффициентами шумоподавления.- invert smoothed values

, and inverted values

are texture dependent noise reduction factors.

вычисляется отдельно для каждого пикселя с координатами

как произведение коэффициента шумоподавления

, заданного пользователем, текстурно-зависимого коэффициента шумоподавления

и зависимости отклонения шума пикселей от яркости пикселей

. Средняя яркость

вычисляется с использованием низкочастотного фильтра, известного из уровня техники.[0117] Estimated pixel noise standard deviation

calculated separately for each pixel with coordinates

as a product of noise reduction

user-defined texture-dependent noise reduction

and Dependence of Pixel Noise Deviation on Pixel Brightness

. Average brightness

calculated using a low-pass filter known in the art.

определяет предпочтения пользователя с точки относительно компромисса между уровнем шумоподавления и уровнем сохранения и восстановления высокочастотных деталей изображения.[0118] Noise reduction ratio

determines the user's preferences from the point regarding the compromise between the level of noise reduction and the level of preservation and restoration of high-frequency image details.

[0119] The determination of noise reduction parameters and the calculation of texture-dependent noise reduction coefficients are performed in the same way for both low-resolution RGB images and low-resolution Bayer images.

[0120] 3DNR for input images in the RGB format is produced in accordance with known methods using noise estimation.

входных изображений низкого разрешения результирующий пиксель

изображений низкого разрешения, подвергнутых 3DNR, вычисляется как сумма взвешенных значений этого пикселя и соседних пикселей, лежащих не далее чем

от текущего пикселя. При этом индивидуальный вес каждого пикселя

, соседнего с

определяется как экспоненциальная функция от квадрата расстояния

между фрагментами изображения низкого разрешения, в которых находятся пиксели

и

, с обратным знаком, деленного на квадрат оцененного значения стандартного отклонения шума пикселей

, а затем делится на сумму индивидуальных весов для всех усредняемых пикселей.[0121] For each pixel

low resolution input images resulting pixel

low-resolution images subjected to 3DNR is calculated as the sum of the weighted values of this pixel and neighboring pixels lying no further than

from the current pixel. In this case, the individual weight of each pixel

adjacent to

defined as an exponential function of the squared distance

between fragments of the low-resolution image in which the pixels are located

and

, with the opposite sign, divided by the square of the estimated value of the standard deviation of pixel noise

, and then divided by the sum of the individual weights for all averaged pixels.

вычисляют как сумму квадратов разностей пикселей, лежащих не дальше

от текущего пикселя

, и соответствующих пикселей, лежащих на таком же расстоянии.[0122] When using the Gaussian noise model, the distance

calculated as the sum of the squares of the differences of pixels lying no further

from current pixel

, and corresponding pixels lying at the same distance.

вычисляют как сумму функций

от двух аргументов

для пикселей, лежащих не дальше

от текущего пикселя

, и соответствующих пикселей, лежащих на таком же расстоянии от

где

[0123] When using the Poisson model, the distance

calculated as the sum of functions

from two arguments

for pixels no further

from current pixel

, and corresponding pixels lying at the same distance from

Where

могут браться как только из текущего, так и из нескольких входных изображений низкого разрешения, включая изображение низкого разрешения, из которого взят пиксель

.[0124] Pixels

can be taken both from the current and from several low-resolution input images, including a low-resolution image from which a pixel is taken

.

учитываются только пиксели, принадлежащие одному и тому же цветовому каналу.[0125] The implementation of 3DNR for Bayer images differs in that only pixels belonging to the same color channel are averaged. The 3DNR implementation for Bayer images is also different in that when calculating the distance

only pixels belonging to the same color channel are taken into account.

используют только пиксели, принадлежащие тому же цветовому каналу, что и текущий пиксель. В одной реализации для вычисления средней яркости можно использовать фильтрацию с фильтром

, с последующим сглаживанием с помощью фильтра в форме диска, известного из уровня техники.[0126] The implementation of 3DNR for Bayer images is different in that when calculating the average brightness

use only pixels that belong to the same color channel as the current pixel. In one implementation, filtering with a filter can be used to calculate average brightness

, followed by smoothing using a filter in the form of a disk, known from the prior art.

[0127] The validity map estimator 107 generates a validity map using a set of low-resolution images and sets of pixel offsets, the reliability map representing the reliability of computing pixel offsets.

[0128] To compensate for possible errors obtained in calculating pixel offsets or changes in the scene at which the difference between adjacent low-resolution images is very large, processing using a validity map is performed. In processing using a confidence map, only pixels whose offsets are calculated reliably are used to compute a high-resolution image.

[0129] The confidence map M has a dimension W x H xk, where W and H are the number of pixels vertically and horizontally, respectively, for a high resolution image to be obtained from low resolution input images having a size of W / sx H / s , and k is the number of input low-resolution images. Each element of the reliability map can take values 1 for pixels whose displacement is calculated reliably, and 0 for pixels whose displacement is calculated unreliably. The displacement of all pixels of the first low-resolution image is assumed to be zero and considered reliable.

[0130] A validity map for low-resolution input RGB images is calculated using the following steps:

- for the input low-resolution images, in addition to the first low-resolution image, pixel offsets are compensated using the calculated pixel offsets and compensated low-resolution images are obtained, the pixel offset being compensated by any method known in the art;

, причем баланс белого, гамма-коррекция, Гауссовская фильтрация выполняется методом, известным в уровне техники;- for all compensated low-resolution images, white balance, gamma correction, Gaussian filtering and conversion to LST color space are performed according to the formulas

moreover, white balance, gamma correction, Gaussian filtering is performed by a method known in the prior art;

;- calculate the pixel-by-pixel Euclidean distance between the pixels in the first low-resolution image in the LST space and the pixels of the compensated low-resolution images in the LST space, filter the calculated Euclidean distances using a low-pass filter, known in the prior art, and perform non-linear transformation according to the equation:

;

- compute the pixel-by-pixel SSIM index (structural similarity index) between the first low-resolution image in the LST space and each of the compensated low-resolution images in the LST space, the number of which is k-1, and set the SSIM index to 0 for those pixels for which the index value SSIM is negative;

- calculate for each pixel of the compensated low-resolution images in the LST space the quotient of the product of the filtered, nonlinearly converted Euclidean distance and the SSIM index and the sum of the filtered, nonlinearly converted Euclidean distance and the SSIM index to obtain an intermediate map consisting of quotient values calculated for pixels of the compensated low resolution in LST space, and values for pixels of the first low-resolution image, each of which is assigned 1 moreover, the resulting intermediate map has a size of W / s x H / s x k;

- scale the intermediate map so as to make its size equal to W x H x k;

- calculate the confidence map by threshold transformation of the scaled intermediate map, while the values of the quotient exceeding the threshold are assigned 1, the remaining values of the quotient are assigned 0, and the values for the pixels of the first low-resolution image are left at 1. The threshold is selected by the user and reflects the user's perception of which errors in calculating pixel offsets are considered unacceptable.

[0131] Calculation of a confidence map for Bayer images is characterized in that compensation for pixel offsets is performed using the following steps:

- divide the pixels of all input low-resolution images into separate color channels G1, G2, R and B, as shown in FIG. 5 as a non-limiting example;

- summarize the pixels of the color channels G1 and G2 with the same positions in all input low-resolution images and divide the resulting sums of pixels of the color channels G1 and G2 by 2 to obtain channel G.

[0132] The rest of the operations for calculating the confidence map for Bayer images are the same as for images in RGB format.

[0133] The texture direction estimation unit 105 determines the direction of the texture in the low-resolution, low-noise images obtained by the preprocessing unit 106.

[0134] The filter selection unit 102 selects either sets of non-directional filters or sets of directional filters based on a confidence map, sets of fractional pixel offsets and texture directions in low-resolution images with reduced noise.

[0135] The high-resolution image acquisition unit 104 obtains high-resolution images in RGB format by computing pixels of a high-resolution image using a confidence map, a set of low-resolution, low-noise images, selected filter sets, and sets of integer pixel offsets.

[0136] The operations of acquiring a high-resolution image using a validity map and a set of low-resolution images with reduced noise (subjected to 3DNR) are performed by the high-resolution image acquiring unit 104 in the same way for both low-resolution input RGB images and low-resolution input Buyer images with taking into account the differences described above in the detailed description of obtaining a high resolution image by the high resolution image obtaining unit 104.

[0137] To filter the color artifacts remaining in the high-resolution image, the super-resolution data processing apparatus 100 further comprises a post-processing unit 108. An embodiment of a super-resolution data processing apparatus 100 comprising a post-processing unit 108 is for processing Bayer images only.

[0138] The post-processing unit 108 converts the high-resolution RGB image obtained by the high-resolution image acquiring module into an YUV image. Then, the post-processing unit 108 processes the YUV image by: calculating a map of standard deviations of pixels in channel Y using a sliding window method, calculating a reference channel Ref for cross-bilateral filtering by smoothing a map of standard deviations of pixels using Gaussian filtering, and performing cross-channel filtering of channels U and V using the reference channel Ref. After processing the image in the YUV format, the post-processing unit 108 inversely converts the processed image in the YUV format to the post-processed high-resolution image in the RGB format.

[0139] The proposed super-resolution data processing method is for obtaining a high resolution image from a set of low resolution images of one scene. The method of processing data for super-resolution can be performed both with input images in RGB format and with input Bayer images, and a demoicing operation for converting Bayer images into images in RGB format is not required. The method of processing data for super-resolution is performed by the device 100 for processing data for super-resolution described above. The operations performed by the blocks of the super-resolution data processing apparatus 100 are fully consistent with the operations performed in the steps of the super-resolution data processing method. Since the operations performed by the units of the super-resolution data processing apparatus 100 are described in detail above, a detailed description of the operations contained in the steps of the super-resolution data processing method is omitted.

[0140] In one embodiment, the super-resolution data processing method comprises the steps of: receiving a set of low-resolution images of one scene; get sets of pixel offsets for low-resolution images for all pixels corresponding to the same fragments in the set of low-resolution images by calculating pixel offsets for pixels corresponding to the same fragments in low-resolution images relative to pixels corresponding to the same fragments in the first a low resolution image selected from a set of low resolution images; obtaining sets of integer pixel offsets by calculating integer pixel offsets for each pixel offset, wherein the sets of integer pixel offsets correspond to sets of pixel offsets for low resolution images; obtaining sets of fractional pixel offsets by calculating fractional pixel offsets for each pixel offset, wherein the sets of fractional pixel offsets correspond to sets of pixel offsets for low resolution images; selecting filter sets from pre-calculated filter sets based on sets of fractional pixel offsets; obtaining a high-resolution image in RGB format by computing pixels of a high-resolution image using a set of low-resolution images, selected filter sets, and sets of integer pixel offsets.

[0141] In another embodiment, the super-resolution data processing method may further include the steps of pre-calculating filter sets that specify a finite number of fractional pixel offsets; generating sets of fractional pixel offsets by selecting all combinations of fractional pixel offsets; and calculating filter sets based on sets of fractional pixel shifts and a regularizer describing local constraints for superresolution.

[0142] In yet another embodiment of the super-resolution data processing method, the directivity of the image texture when taking a high resolution image from a set of low resolution images of one scene is taken into account. In this embodiment, the super-resolution data processing method further comprises determining the texture directions of the low-resolution images, wherein the pre-computed filter sets include omnidirectional filter sets and directional filter sets, and the selection of filter sets contains either a selection of non-directional filter sets, or sets of directional filters based on sets of fractional pixel offsets and texture directions in low-resolution images. Omnidirectional filters are associated with image areas containing an undirected texture, and directional filters are associated with image areas containing an undirected texture.

[0143] In yet another embodiment of the super-resolution data processing method, additional restrictions are taken into account, such as noise present in input low-resolution images, inaccurate estimation of pixel offsets in some areas of low-resolution images, or objective impossibility to estimate pixel offsets due to a sharp change in images , for example, caused by a change in lighting, as well as the presence of residual color artifacts in the output image.

[0144] In this embodiment, the superresolution data processing method further comprises the steps of: determining noise reduction parameters for all pixels in low resolution images, including a noise model, image fragment size, search radius for adjacent image fragments, and standard deviation pixel noise from pixel brightness; calculating texture dependent noise reduction coefficients based on changes in pixel brightness, wherein each texture dependent noise reduction coefficient is calculated for each pixel of low resolution images; evaluating the standard deviation of the pixel noise using the dependence of the standard deviation of the pixel noise on the pixel brightness, texture-dependent noise reduction coefficients and the noise reduction coefficient specified by the user; performing three-dimensional noise reduction (3DNR) on a set of low-resolution images based on noise reduction parameters and estimated pixel noise standard deviation values to obtain low-resolution images with reduced noise; and generate a confidence map using a set of low-resolution images and sets of pixel offsets. The confidence map represents the reliability of the calculation of pixel offsets, while determining the directions of the texture of low-resolution images includes determining the directions of the texture of low-resolution images with reduced noise.

[0145] In this embodiment, the selection of filter sets comprises a selection of either sets of non-directional filters or sets of directional filters based on a confidence map, sets of fractional pixel offsets and texture directions in low-resolution images with reduced noise; however, obtaining a high-resolution image in RGB format includes computing pixels of a high-resolution image using a confidence map, a set of low-resolution images with reduced noise, selected filter sets and sets of integer pixel offsets.

[0146] To filter color artifacts remaining in the high-resolution image, the super-resolution data processing method further comprises the steps of: converting the high-resolution image in RGB format to an image in YUV format; process the image in YUV format; and perform the inverse conversion of the processed image in YUV format to the post-processed high-resolution image in RGB format.

[0147] Image processing in YUV format is performed by: calculating a map of standard deviations of pixels in channel Y by a sliding window method, calculating a reference channel Ref by smoothing a map of standard deviations of pixels using Gaussian filtering and cross-channel filtering of channels U and V using a reference channel Ref.

[0148] The present invention can be applied to various image capture systems with one or more cameras, including a system supporting a burst function. The present invention can be used both as “digital zoom” and for enlarging image megapixels (Mpx) in a common camera use case. The present invention can be used in applications requiring optical character recognition (OCR). The present invention can be implemented as a single-chip system (SoC), as software using a general purpose processor, digital signal processor (DSP) and / or central processing unit (CPU), or partially as SoC, and partially as software. In addition, the present invention can be implemented as a separate device or part of an image signal processor (ISP).

[0149] The present invention should not be limited to the above examples of the use and implementation of the present invention, and it should be obvious to a person skilled in the art that other types of use and implementation of the present invention can be used to improve image and video quality.

[0150] The above description of embodiments of the invention is illustrative, and modifications to the configuration and implementation are within the scope of the present description. For example, although embodiments of the invention have generally been described with respect to FIG. 1-3, these descriptions are exemplary. Although the subject of patenting has been described in a language specific for design features or methodological actions, it is understood that the subject of patenting defined in the attached claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and acts described above are disclosed as examples of forms for implementing the claims. In addition, the invention is not limited to the illustrated order of the steps of the method, the order can be changed by a person skilled in the art without creative efforts. Some or all of the steps of the method may be performed sequentially or simultaneously.

Claims (72)

1. A device for processing data for superresolution, containing: an offset estimation unit configured to: take a set of low-resolution images of one scene, obtain sets of pixel offsets for low-resolution images for all pixels corresponding to the same fragments in the set of low-resolution images by calculating pixel offsets for pixels corresponding to the same fragments in low-resolution images relative to pixels corresponding to the same fragments in the first a low resolution image selected from a set of low resolution images, obtain sets of integer pixel offsets by calculating integer pixel offsets for each pixel offset, wherein sets of integer pixel offsets correspond to sets of pixel offsets for low resolution images, obtain sets of fractional pixel offsets by calculating fractional pixel offsets for each pixel offset, wherein the sets of fractional pixel offsets correspond to sets of pixel offsets for low resolution images; a filter bank configured to store filter sets, the filter sets being pre-computed; a filter selection unit configured to select filter sets from a filter bank based on sets of fractional pixel offsets; a high-resolution image acquisition unit configured to obtain a high-resolution image in RGB format by computing pixels of a high-resolution image using a set of low-resolution images, selected filter sets, and sets of integer pixel offsets. 2. The device according to claim 1, in which the pre-calculated filter sets are obtained by: setting a finite number of fractional pixel offsets; generating sets of fractional pixel offsets by selecting all combinations of fractional pixel offsets; computing filter sets based on sets of fractional pixel shifts and a regularizer describing local constraints for superresolution. 3. The device according to claim 1, in which the number of filters stored in the filter bank is reduced based on at least one of the conditions for matching pixel offsets. 4. The device according to claim 1, wherein the sets of filters stored in the filter bank include sets of non-directional filters and sets of directional filters, moreover, non-directional filters are associated with areas of the image containing the non-directional texture, and directional filters are associated with areas of the image containing the directional texture. 5. The device according to claim 4, further comprising a block for estimating the direction of the texture, configured to determine the direction of the texture of the low-resolution images, moreover, the filter selection unit is further configured to select either sets of non-directional filters or sets of directional filters based on sets of fractional pixel offsets and texture directions in low resolution images. 6. The device according to claim 5, further comprising: a preprocessing unit configured to: determining noise reduction parameters for all pixels in low-resolution images, including the noise model, image fragment size, search radius for adjacent image fragments, and the dependence of the standard deviation of pixel noise on pixel brightness; calculating texture-dependent noise reduction coefficients based on changes in pixel brightness, wherein each texture-dependent noise reduction coefficient is calculated for each pixel of low resolution images; estimating the standard deviations of the pixel noise using the dependence of the standard deviation of the pixel noise on the pixel brightness, texture-dependent noise reduction coefficients and the noise reduction coefficient specified by the user; performing three-dimensional noise reduction (3DNR) on a set of low-resolution images based on noise reduction parameters and estimated pixel noise standard deviation values to obtain low-resolution images with reduced noise; and a validity map estimating unit configured to generate a validity map using a set of low-resolution images and sets of pixel offsets, the reliability map representing the reliability of computing pixel offsets, wherein the unit for evaluating the direction of the texture is further configured to determine the direction of the texture of low-resolution images with reduced noise; wherein the filter selection unit is further configured to select either sets of non-directional filters or sets of directional filters based on a confidence map, sets of fractional pixel offsets and texture directions in low-resolution images with reduced noise; wherein the high-resolution image acquisition unit is further configured to receive the high-resolution image in RGB format by calculating the pixels of the high-resolution image using a confidence map, a set of low-resolution images with reduced noise, selected filter sets and sets of integer pixel offsets. 7. The device according to any one of paragraphs. 1-6, in which the low-resolution images of one scene are images in RGB format. 8. The device according to any one of paragraphs. 1-6, in which the low-resolution images of one scene are Bayer images. 9. The device according to p. 8, further comprising a post-processing unit, configured to: converting a high-resolution image in RGB format obtained by the high-resolution image acquisition module into an image in YUV format; process the image in YUV format by: calculating a map of standard deviations of pixels in channel Y by the sliding window method, computing the reference channel Ref by smoothing the map of standard deviations of pixels using Gaussian filtering, cross-bilateral filtering of channels U and V using the reference channel Ref and reverse converting the processed YUV image to the post-processed high-resolution RGB image. 10. A method for processing data for super-resolution, comprising the steps of: accept a set of low-resolution images of one scene; get sets of pixel offsets for low-resolution images for all pixels corresponding to the same fragments in the set of low-resolution images by calculating pixel offsets for pixels corresponding to the same fragments in low-resolution images relative to pixels corresponding to the same fragments in the first a low resolution image selected from a set of low resolution images; obtaining sets of integer pixel offsets by calculating integer pixel offsets for each pixel offset, wherein the sets of integer pixel offsets correspond to sets of pixel offsets for low resolution images; obtaining sets of fractional pixel offsets by calculating fractional pixel offsets for each pixel offset, wherein the sets of fractional pixel offsets correspond to sets of pixel offsets for low resolution images; selecting filter sets from pre-calculated filter sets based on sets of fractional pixel offsets; obtaining a high-resolution image in RGB format by computing pixels of a high-resolution image using a set of low-resolution images, selected filter sets, and sets of integer pixel offsets. 11. The method according to p. 10, in which pre-computed filter sets are obtained by: setting a finite number of fractional pixel offsets; generating sets of fractional pixel offsets by selecting all combinations of fractional pixel offsets; computing filter sets based on sets of fractional pixel shifts and a regularizer describing local constraints for superresolution. 12. The method of claim 10, wherein the number of said filters is reduced based on at least one of a condition for matching pixel offsets. 13. The method of claim 10, wherein the pre-computed filter sets include omnidirectional filter sets and directional filter sets, moreover, non-directional filters are associated with areas of the image containing the non-directional texture, and directional filters are associated with areas of the image containing the directional texture. 14. The method of claim 13, further comprising the step of: determine the direction of the texture of low-resolution images, however, the selection of filter sets contains a selection of either sets of non-directional filters or sets of directional filters based on sets of fractional pixel offsets and texture directions in low-resolution images. 15. The method of claim 14, further comprising the steps of: determining noise reduction parameters for all pixels in low-resolution images, including the noise model, image fragment size, search radius for adjacent image fragments, and the dependence of the standard deviation of pixel noise on pixel brightness; calculating texture dependent noise reduction coefficients based on changes in pixel brightness, wherein each texture dependent noise reduction coefficient is calculated for each pixel of low resolution images; evaluating the standard deviation of the pixel noise using the dependence of the standard deviation of the pixel noise on the pixel brightness, texture-dependent noise reduction coefficients and the noise reduction coefficient specified by the user; performing three-dimensional noise reduction (3DNR) on a set of low-resolution images based on noise reduction parameters and estimated standard noise deviations of pixels to obtain low-resolution images with reduced noise and generating a confidence map using a set of low-resolution images and sets of pixel offsets, wherein the confidence map represents the reliability of computing pixel offsets, wherein determining the directions of the texture of low-resolution images includes determining the directions of the texture of low-resolution images with reduced noise; wherein the selection of filter sets comprises a selection of either sets of non-directional filters or sets of directional filters based on a confidence map, sets of fractional pixel offsets and texture directions in low-resolution images with reduced noise; however, obtaining a high-resolution image in RGB format includes computing pixels of a high-resolution image using a confidence map, a set of low-resolution images with reduced noise, selected filter sets and sets of integer pixel offsets. 16. The method according to any one of paragraphs. 10-15, in which the low-resolution images of one scene are RGB images. 17. The method according to any one of paragraphs. 10-15, in which the low-resolution images of one scene are Bayer images. 18. The method of claim 17, further comprising the steps of: convert a high-resolution image in RGB format to an image in YUV format; process the image in YUV format by: calculating a map of standard deviations of pixels in channel Y by the sliding window method, computing the reference channel Ref by smoothing the map of standard deviations of pixels using Gaussian filtering, cross-bilateral filtering of channels U and V using the reference channel Ref; and perform the inverse conversion of the processed image in YUV format to the post-processed image of high resolution in RGB format.

Images