CN112862695A - Image restoration method and apparatus - Google Patents
Image restoration method and apparatus Download PDFInfo
- Publication number
- CN112862695A CN112862695A CN202010362078.3A CN202010362078A CN112862695A CN 112862695 A CN112862695 A CN 112862695A CN 202010362078 A CN202010362078 A CN 202010362078A CN 112862695 A CN112862695 A CN 112862695A
- Authority
- CN
- China
- Prior art keywords
- image
- target image
- target
- pixel position
- position information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000015556 catabolic process Effects 0.000 claims description 46
- 238000006731 degradation reaction Methods 0.000 claims description 46
- 238000013528 artificial neural network Methods 0.000 claims description 18
- 230000015654 memory Effects 0.000 claims description 12
- 230000004044 response Effects 0.000 claims description 9
- 230000004075 alteration Effects 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims 1
- 238000000844 transformation Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 description 21
- 238000010586 diagram Methods 0.000 description 16
- 230000006870 function Effects 0.000 description 16
- 230000008569 process Effects 0.000 description 8
- 230000002093 peripheral effect Effects 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 238000012549 training Methods 0.000 description 6
- 230000006866 deterioration Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 230000004913 activation Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 150000001875 compounds Chemical class 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 241000238631 Hexapoda Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
Images
Classifications
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G06T5/60—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G06T3/18—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
- G06T3/4076—Super resolution, i.e. output image resolution higher than sensor resolution by iteratively correcting the provisional high resolution image using the original low-resolution image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G06T5/80—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
An image restoration method and apparatus are provided. The image restoration method includes: acquiring a target image; a restored image of the target image is acquired from an image restoration model into which the target image and pixel position information of the target image are input.
Description
This application is based on and claims priority from korean patent application No. 10-2019-0155506, filed by the korean intellectual property office at 28.11.2019, the disclosure of which is incorporated herein by reference in its entirety.
Technical Field
One or more example embodiments provide methods and apparatuses related to a method and apparatus for restoring an image.
Background
Recently, due to the development of optical technology and image processing technology, image photographing apparatuses are used in a wide range of fields (e.g., a multimedia content field, a security field, or an identification field). For example, the image photographing apparatus may be installed in a mobile device, a camera, a vehicle, or a computer to photograph an image, recognize an object, or acquire data for controlling a device. However, since a portion having degradation is included in an image captured by an image capturing apparatus, a method of removing the degradation is required.
Disclosure of Invention
One or more example embodiments may address at least the above problems and/or disadvantages and other disadvantages not described above. Furthermore, example embodiments need not overcome the disadvantages described above, and example embodiments may not overcome any of the problems described above.
According to one aspect of the disclosure, there is provided an image restoration method including: acquiring a target image; and obtaining a restored image of the target image from the image restoration model based on the target image and the pixel position information of the target image.
The pixel location information may include an image representing pixel locations in the target image based on two-dimensional (2D) coordinates.
The pixel position information may include a first image in which the first value changes in a first axis direction in cartesian coordinates and a second image in which the second value changes in a second axis direction perpendicular to the first axis direction, and the first image and the second image have the same resolution as the target image.
The pixel position information may include a third image in which the third value is changed based on a distance from the reference point in the polar coordinates, and a fourth image in which the fourth value is changed based on an angle with respect to the reference line, and the third image and the fourth image have the same resolution as the target image.
The reference point may be a principal point of a lens taking the target image or a central point of the target image.
The pixel position information may include a first image in which a first value is changed in a first axis direction in cartesian coordinates, a second image in which a second value is changed in a second axis direction perpendicular to the first axis direction, a third image in which a third value is changed based on a distance from a reference point in polar coordinates, a fourth image in which a fourth value is changed based on an angle with respect to the reference line, and the first image, the second image, the third image, and the fourth image have the same resolution as the target image.
The pixel position information and the target image may be coupled to each other and input to the image restoration model.
The pixel location information and the pixel values of the target image may be input to the convolution layer of the image restoration model.
The target image may include different levels of degradation based on pixel location.
The degradation may be caused by aberrations of a lens used to capture the target image.
The target image may include at least one Low Resolution (LR) image having different levels of degradation based on pixel location, and the restored image is a High Resolution (HR) image with reduced degradation relative to the at least one LR image.
The image restoration model may be trained as: a reference restored image having reduced degradation with respect to a reference target image is output in response to input of the reference target image having different levels of degradation based on the pixel position and reference pixel position information of the reference target image.
According to another aspect of the disclosure, there is provided a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image restoration method.
According to another aspect of the disclosure, there is provided an image restoration method including: acquiring a plurality of target images shot based on a multi-lens array; and obtaining a restored image of the plurality of target images from an image restoration model based on the plurality of target images and pixel position information of the plurality of target images.
The multi-lens array may include a plurality of lens elements, and the step of obtaining the restored image includes: a restored image is obtained from an image restoration model based on positional information of a lens element that captured each of the plurality of target images.
According to another aspect of the disclosure, there is provided an image restoration apparatus including: at least one processor configured to: acquiring a target image; and obtaining a restored image of the target image from the image restoration model based on the target image and the pixel position information of the target image.
The pixel location information may include an image representing pixel locations in the target image based on two-dimensional (2D) coordinates.
The pixel position information may include a first image in which the first value changes in a first axis direction in cartesian coordinates and a second image in which the second value changes in a second axis direction perpendicular to the first axis direction, and the first image and the second image have the same resolution as the target image.
The pixel position information may include a third image in which the third value is changed based on a distance from the reference point in the polar coordinates, and a fourth image in which the fourth value is changed based on an angle with respect to the reference line, and the third image and the fourth image have the same resolution as the target image.
The pixel position information and the target image may be coupled to each other and input to the image restoration model.
According to another aspect of the disclosure, there is provided a computer-implemented method of restoring images in a trained neural network, the method comprising: receiving a target image; receiving pixel position information; applying one or more transforms to the target image based on pixel location information in one or more layers of the trained neural network; creating a restored image based on the one or more transforms applied to the target image; and outputting the restored image.
According to another aspect of the disclosure, there is provided an image restoration apparatus including: a memory storing one or more instructions; and at least one processor configured to execute the one or more instructions to: receiving a target image; receiving pixel position information; applying one or more transforms to the target image based on pixel location information in one or more layers of the trained neural network; creating a restored image based on the one or more transforms applied to the target image; and outputting the restored image.
Drawings
The above and/or other aspects will become more apparent by describing example embodiments with reference to the attached drawings, in which:
FIG. 1A is a diagram illustrating an image restoration model according to an example embodiment;
fig. 1B and 1C are diagrams illustrating a lens for photographing a target image according to an example embodiment;
FIG. 2 is a diagram illustrating degradation of a target image according to an example embodiment;
FIG. 3 is a diagram illustrating operation of an image restoration device according to an example embodiment;
fig. 4 and 5 are diagrams illustrating pixel position information according to example embodiments;
FIG. 6 is a diagram illustrating a process in which pixel position information and a target image are input to a convolutional layer according to an example embodiment;
FIG. 7 is a diagram illustrating an apparatus capable of implementing an image restoration apparatus according to an example embodiment;
FIG. 8 is a flowchart illustrating an image restoration method according to an example embodiment; and
fig. 9 is a diagram illustrating an image restoration apparatus according to an example embodiment.
Detailed Description
Example embodiments will be described in detail with reference to the drawings, wherein like reference numerals refer to like elements throughout.
The following description of structure or function is merely an example for describing example embodiments, and the scope of example embodiments is not limited to the description provided in this specification. Various changes and modifications to the example embodiments may be practiced by those of ordinary skill in the art.
Although the terms "first" or "second" are used to explain various components, the components are not limited to the terms. These terms are only used to distinguish one component from another component. For example, a "first" component may be referred to as a "second" component, or, similarly, a "second" component may be referred to as a "first" component, within the scope of rights in accordance with the concepts of the present disclosure.
It should be noted that if the specification describes a first component "connected," "coupled," or "joined" to a second component, a third component may be "connected," "coupled," or "joined" between the first and second components, although the first component may be directly connected, coupled, or joined to the second component.
As used herein, the singular forms also are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components, and/or groups thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Unless otherwise defined herein, all terms (including technical or scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. Unless otherwise defined herein, terms defined in general dictionaries should be interpreted as having a meaning that matches the contextual meaning in the relevant art and will not be interpreted in an idealized or overly formal sense.
With respect to the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be designated by the same reference numerals as much as possible even if the same elements are shown in different drawings.
Fig. 1A is a diagram illustrating an image restoration model according to an example embodiment.
Referring to fig. 1A, the image restoration model 130 may output a restored image 140 based on the pixel position information 120 in response to an input of the target image 110. The target image 110 may represent an image input to the image restoration model 130 and may be an image including different levels of degradation based on pixel location. The degradation may be caused by aberrations of a lens used to capture the target image 110. The lens may be a multi-lens array or a single lens, as will be further described below with reference to fig. 1B and 1C. The restored image 140 may represent an image in which the degradation of the target image 110 is reduced, and may be output from the image restoration model 130.
The image restoration model 130 may perform image processing of the input target image 110 and may output a restored image 140. Image processing may include, for example, Super Resolution (SR), deblurring, denoising, demosaicing, or image inpainting. The SR may be image processing for increasing a resolution of an image, the deblurring may be image processing for removing blur included in the image, the denoising may be image processing for removing noise included in the image, the demosaicing may be image processing for reconstructing a full-color image from an incomplete color sample, and the image inpainting may be image processing for reconstructing a lost or damaged portion of the image.
The pixel location information 120 may be based on at least one image representing pixel locations in the target image 110 according to two-dimensional (2D) coordinates. The image representing the pixel location information 120 may have the same resolution as the target image 110, and thus, the image representing the pixel location information 120 may indicate the location of the pixel included in the target image 110, as will be further described below with reference to fig. 4 and 5.
Fig. 1B and 1C are diagrams illustrating a lens for photographing a target image according to an example embodiment.
Referring to fig. 1B, a plurality of target images may be photographed based on the multi-lens array 150.
The multi-lens array 150 may include a plurality of lens elements 151a, 151b, 151c, and 151 d. When the size of each lens element included in multi-lens array 150 is reduced, focal length f of each lens element (e.g., lens element 151a)1And the thickness of the image photographing apparatus can be reduced. That is, when the number of lens elements included in the same region increases, the focal length f of the lens element 151a1And the thickness of the image photographing apparatus can be reduced. As described above, a thin camera can be realized by reducing the size of each lens element included in the multi-lens array 150. In another example, the lens element 151a may also be implemented in a multi-layer structure.
The individual lens elements 151a of the multi-lens array 150 may cover an area of the sensor 160 corresponding to the size of the individual lens elements 151 a. The area of the sensor 160 may be predetermined. For example, light 155 passing through individual lens elements 151a may be incident on sensing elements included in the region of sensor 160. Light 155 may include a plurality of light rays. Light ray 156 may correspond to a stream of photons 157. Each sensing element included in sensor 160 may generate sensed information based on light rays 156 passing through lens elements of multi-lens array 150. For example, the sensing element 161 may generate sensing information based on the light rays 156 incident on the lens element 151 a. Further, light passing through another separate lens element 151b may be incident on a portion of the sensing element in addition to light 155 depicted by light ray 156.
Light passing through the plurality of lens elements may be sensed and a Compound Eye Vision (CEV) image 170 may be acquired as the target image. The CEV image 170 may represent an image acquired by overlappingly capturing the same scene with slightly different viewpoints in a manner similar to the operation of a compound eye of an insect. For example, the CEV image 170 may be acquired based on the intensity of light received by the plurality of sensing elements through the plurality of lens elements arranged in an array. The images included in the CEV image 170 may correspond to a plurality of lens elements, respectively.
The CEV image 170 may be input to the image restoration model as the target image, and the CEV image 170 and pixel position information of the individual images included in the CEV image 170 may be input to the image restoration model. The individual images included in the CEV image 170 may have the same resolution, and the pixel location information may also be input to the image restoration model as a single frame having the same resolution as the resolution of the individual images included in the CEV image 170. The image restoration model may determine a restored image having a high resolution from the input CEV image 170 having a low resolution.
Referring to fig. 1C, at least one target image 190 may be photographed based on a single lens 180. The target image 190 may be captured based on a single lens 180. Although the single lens 180 may have a larger size than each of the individual lens elements included in the multi-lens array 150 of fig. 1BSize, and may have a relatively long focal length f based on size2However, the volume of the image capturing apparatus may also increase. When one photographing is performed using a single lens 180, a single target image (i.e., a target image 190) may be acquired. When the same scene is continuously photographed several times, a plurality of target images can be acquired. The acquired plurality of target images and pixel position information may be input to an image restoration model.
Fig. 2 is a diagram illustrating degradation of a target image according to an example embodiment.
Referring to fig. 2, the target image 210 includes different levels of degradation based on pixel location. For convenience of description, the target image 210 may represent a plurality of individual images included in a CEV image captured by a multi-lens array or a single image captured by a single lens, and the degradation characteristics are described based on the central portion 220 and the peripheral portion 230 of the target image 210. As shown in fig. 2, stronger degradation may occur in the peripheral portion 230 than in the central portion 220, which may be caused by an increase in the degree of degradation from the central portion of the image to the peripheral portion of the image due to lens aberration. The above degradation characteristics may also be verified in the Point Spread Function (PSF) 240. In the PSF 240, execution is concentrated at a point in the central portion 220 rather than in the peripheral portion 230. Accordingly, a first portion closer to the central portion 220 of the target image 210 has weaker degradation than a second portion farther from the central portion. Further, the degradation may have a symmetrical shape (e.g., axial symmetry) based on the center of the lens.
Fig. 3 is a diagram illustrating an operation of an image restoration apparatus according to an example embodiment.
Fig. 3 shows an example of a process in which the image restoration apparatus acquires a restored image 340 from an image restoration model 330 into which a target image 310 and pixel position information 320 of the target image 310 are input.
The target image 310 may represent at least one image having different levels of degradation based on pixel location, and may be, for example, a plurality of Low Resolution (LR) images. The plurality of LR images may be a plurality of individual images included in the CEV image captured by the multi-lens array, or a plurality of images acquired by sequentially capturing the same scene several times using a single lens. Even if the same scene is continuously and rapidly photographed using a single lens, information included in a photographed image may slightly change due to a minute movement of an image sensor, and a single high-resolution (HR) image may be obtained by combining the above various information. Even when the same information is included in a plurality of LR images, an image in which noise in the LR image is reduced can be obtained by averaging the information of the LR image.
The pixel location information 320 may include information of the location of the pixels included in the target image 310. The pixel location information 320 will be further described below with reference to fig. 4 and 5.
The target image 310 and the pixel location information 320 may be coupled or associated with each other and may be input to the image restoration model 330. The image restoration model 330 may be a neural network that processes the input image using at least one convolutional layer. The above-described target image 310 and the pixel position information 320 may be coupled to each other and may be input to the convolution layer 331 included in the image restoration model 330. When the pixel position information 320 indicating the position of each pixel in the image and the target image 310 are input to the image restoration model 330, the image restoration model 330 may apply an image restoration function suitable for the pixel to be restored by using the position information of the pixel. Therefore, the recovery performance can be effectively enhanced.
The neural network may correspond to an example of a Deep Neural Network (DNN). The DNN may include, for example, a fully connected network, a deep convolutional network, or a Recurrent Neural Network (RNN). The neural network may perform image restoration by mapping input data and output data in a nonlinear relationship based on deep learning. Deep learning represents a machine learning scheme for solving problems such as image restoration from large data sets. The input data and the output data may be mapped to each other by supervised learning or unsupervised learning among the deep learning.
The neural network may include an input layer, a hidden layer, and an output layer. Each of the input layer, the hidden layer, and the output layer may include a plurality of nodes.
According to an example embodiment, nodes of a layer other than an output layer in a neural network may be connected to nodes of a next layer via links to transmit output signals. The plurality of links may correspond to a plurality of nodes included in a next layer. The layers other than the output layer may be an input layer and a plurality of hidden layers.
According to an example embodiment, an output of an activation function associated with a weighted input of a node included in a previous layer may be input to each node included in a hidden layer. The weighted input may be obtained by multiplying a weight with an input of a node included in a previous layer. The weights may be referred to as parameters of the neural network. For example, the weights may correspond to kernel elements in a kernel matrix included in the convolutional layer. The activation function may include, for example, a sigmoid function, a hyperbolic tangent (tanh) function, or a modified linear unit (ReLU) function. The non-linearity may be formed in the neural network by an activation function. The weighted input of the nodes included in the previous layer may be input to each node included in the output layer.
When the width and depth of the neural network are sufficiently large, the neural network may have a capacity large enough to realize an arbitrary function. When the neural network learns a sufficiently large amount of training data through an appropriate learning process, an optimal image restoration performance can be achieved.
The image restoration model 330 may output a restored image 340 in response to the input of the target image 310 and the pixel location information 320. The restored image 340 may represent an image in which the deterioration included in the target image 310 is reduced or removed, and may include, for example, an HR image.
Fig. 4 and 5 are diagrams illustrating pixel position information according to example embodiments.
Fig. 4 illustrates pixel position information indicating positions of pixels included in a target image based on Cartesian (Cartesian) coordinates.
The pixel position information may include a first image 410 whose values change in one axis direction in cartesian coordinates and a second image 420 whose values change in another axis direction perpendicular to the one axis direction. The first image 410 and the second image 420 may have the same resolution as the resolution of the target image. For example, the first image 410 may be an image having a value changed in the x-axis direction although having the same value in the y-axis direction, and the second image 420 may be an image having a value changed in the y-axis direction although having the same value in the x-axis direction. The image restoration model may verify x-axis values of corresponding pixel locations from the first image 410, may verify y-axis values of corresponding pixel locations from the second image 420, and may utilize the corresponding pixel location (x, y) during image restoration.
Fig. 5 illustrates pixel position information indicating positions of pixels included in a target image based on polar coordinates.
The pixel position information may include a third image 510 and a fourth image 520, the value of the third image 510 being changed based on a distance from a reference point in polar coordinates, and the value of the fourth image 520 being changed based on an angle with respect to the reference line. The third image 510 and the fourth image 520 may have the same resolution as that of the target image. The third image 510 may be an image that: the value of the image changes in response to a change in distance from the reference point, but when the distance from the reference point is the same, the value of the image has the same value despite the change in angle with respect to the reference line. For example, the third image 510 may have the lowest value at the reference point and the highest value at the four vertices of the third image 510 that are farthest from the reference point. Further, the fourth image 520 may be an image: the value of the image changes in response to a change in the angle with respect to the reference line, but when the angle with respect to the reference line is the same, the value of the image has the same value although the distance from the reference point changes. For example, the fourth image 520 may have the lowest value in the reference line and the highest value at the largest angle in a predetermined direction (e.g., clockwise or counterclockwise) with respect to the reference line. Although the angle with respect to the reference line is divided into "16" parts as shown in fig. 5 for convenience of description, example embodiments are not limited thereto. For example, the angle with respect to the reference line may be divided into "n" pieces, and n may be a natural number greater than or equal to "1". The image restoration model may verify the distance r between the corresponding pixel and the reference point from the third image 510, may verify the angle θ with respect to the reference line of the corresponding pixel from the fourth image 520, and may utilize the position (r, θ) of the corresponding pixel during image restoration.
In this example, the reference point may be a principal point (principal point) of a lens that captures the target image. The principal point may be a point at which the optical axis and the principal plane of the lens intersect each other, and may represent the center of the lens. The principal point may represent the optical center of the lens and may be different from the geometric center of the lens. In another example, the reference point may be the center of the target image. According to example embodiments, when the principal point of the lens is not verified, the center of the target image may be set as a reference point.
In one example, the pixel location information input to the image restoration model may be based on 2D coordinates determined from, for example, the image restoration model or the target image. In one example, because the blur caused by the lens is generally radially distributed from the center of the lens, when the image restoration model removes the blur included in the target image, pixel position information based on polar coordinates rather than cartesian coordinates may be selected and input to the image restoration model. In another example, when the degradation included in the target image changes vertically and/or horizontally, when only a central portion of an image photographed by the image sensor is cropped and input to the image restoration model as the target image, pixel position information based on cartesian coordinates rather than polar coordinates may be more useful for image restoration.
In another example, the pixel location information may include an image indicating locations of pixels in the target image based on cartesian coordinates and an image indicating locations of pixels in the target image based on polar coordinates. In this example, the pixel position information may include the first and second images 410 and 420 of fig. 4 and the third and fourth images 510 and 520 of fig. 5, and the image restoration model may perform image restoration based on the pixel position information according to cartesian coordinates and/or polar coordinates according to the deterioration of the target image to be restored.
Fig. 6 is a diagram illustrating a process in which pixel position information and a target image are input to a convolutional layer according to an example embodiment.
Fig. 6 shows a target image 610 and pixel location information 620 concatenated with each other.
The target image 610 may include a plurality of LR images, and each LR image may include three feature images (e.g., a red (R) image, a green (G) image, and a blue (B) image). The target image 610 may include, for example, a plurality of individual images included in a CEV image captured by a multi-lens array or a single image captured by a single lens. As described above, pixel location information 620 may include two images that indicate pixel locations in target image 610 based on 2D coordinates.
The target image 610 and the pixel location information 620 may be coupled to each other and may be input to a convolution layer of the image restoration model. Two position information images and a plurality of LR images corresponding to each region may be input to the convolution layer. For example, two position information images and a plurality of feature images corresponding to the peripheral portion 630 may be input together to the convolutional layer, and two position information images and a plurality of feature images corresponding to the central portion 640 may be input together to the convolutional layer, and thus, when image processing is performed in the image restoration model, an optimal image restoration function may be applied to each region based on the position of the corresponding region, in the hope of more efficient image restoration. The image restoration function may represent at least one operation applied to the target image 610 in the image restoration model to obtain a restored image, and the image restoration function applied to the corresponding region may vary according to the position of the region in the target image 610. For example, different levels of degradation may occur in the peripheral portion 630 and the central portion 640 of the target image 610, and thus different image restoration functions may be applied to the peripheral portion 630 and the central portion 640. The at least one operation may be an operation suitable for an image restoration model, and may include, for example, a convolution operation, a ReLU operation, or a pooling operation.
An image restoration model that reduces different levels of degradation for each pixel region using the above image restoration function may be predetermined through machine learning. The image restoration model may be a model that is trained in advance to output a reference restored image with reduced degradation in response to input of reference target images with different levels of degradation based on pixel positions and reference pixel position information of the reference target images. The reference target image used during training may include deterioration based on characteristics of a lens that captures the target image to be actually restored, and may have the same resolution as that of the target image to be actually restored. Further, the reference pixel position information used during training may be the same as the pixel position information used for actual image restoration.
As described above, the reference target image, the reference pixel position information, and the reference restored image can be used as training data, and the image restoration model can be trained even if blur information based on the lens design is missing.
Different levels of degradation based on pixel position may occur for each lens capturing a target image, and the size of data input to the image restoration model (e.g., the size of pixel position information) may be changed based on the resolution of the target image, and thus, the image restoration model may be trained separately based on the lens capturing the target image and the resolution of the target image. For example, an image recovery model that removes degradation of a target image captured by a smartphone camera and an image recovery model that removes degradation of a target image captured by a normal camera may be models trained based on different pieces of training data and may be distinguished from each other.
Fig. 7 is a diagram illustrating an apparatus capable of implementing an image restoration apparatus according to an example embodiment.
The image restoration apparatus can be applied to remove degradation of images photographed by various devices. Further, the image restoration apparatus may be applied to use an image that does not include degradation in various technical fields (e.g., object recognition, style migration, or unsupervised image generation).
For example, as shown in fig. 7, an image restoring apparatus may be installed in the mobile device 700 to restore an image photographed by a camera in the mobile device 700. Although three cameras 710, 720, and 730 are installed for convenience of description as shown in fig. 7, various numbers of cameras may be included in the mobile device 700. Cameras 710, 720, and 730 may be different cameras and have different lens specifications (e.g., wide angle lens, standard lens, telephoto lens, or multiple lenses), or have different resolutions. The image restoration apparatus may remove degradation of an image captured by each of the cameras 710, 720, and 730 using three image restoration models. In one example, a single image restoration model may be trained through multitask learning to remove degradation of images captured by each of the cameras 710, 720, and 730. In this example, information indicating which camera among the cameras 710, 720, and 730 takes the target image to be restored may be additionally input to the image restoration model.
Fig. 8 is a flowchart illustrating an image restoration method according to an example embodiment.
Fig. 8 illustrates an image restoration method performed by a processor included in an image restoration apparatus according to an example embodiment.
In operation 810, the image restoration apparatus acquires a target image. For example, the image restoration apparatus may acquire a plurality of individual images included in a CEV image captured by a multi-lens array or a single image captured by a single lens as the target image. The pixel location information may indicate a pixel location in the target image based on the 2D coordinates. In one example, the pixel position information may include a first image whose value changes in one axis direction in cartesian coordinates and a second image whose value changes in another axis direction perpendicular to the one axis direction, and the first image and the second image may have the same resolution as that of the target image. In another example, the pixel position information may include a third image whose value changes based on a distance from a reference point in polar coordinates and a fourth image whose value changes based on an angle with respect to the reference line, and the third image and the fourth image have the same resolution as that of the target image. In another example, the pixel position information may include a first image whose value changes in one axis direction in cartesian coordinates, a second image whose value changes in another axis direction perpendicular to the one axis direction, a third image whose value changes based on a distance from a reference point in polar coordinates, and a fourth image whose value changes based on an angle with respect to a reference line.
In operation 820, the image restoration apparatus acquires a restored image of the target image from the image restoration model into which the target image and the pixel position information of the target image are input. The pixel position information and the target image may be coupled to each other and input to a convolution layer of the image restoration model. The target image may be an image having different levels of degradation based on pixel positions, and the restored image may be an image with reduced degradation.
The image restoration model may be a model trained to output a reference restored image with reduced degradation in response to input of a reference target image with different levels of degradation based on pixel positions and reference pixel position information of the reference target image.
According to example embodiments, when a target image corresponds to a plurality of individual images included in a CEV image captured by a multi-lens array, position information, identification information, or arrangement information of a plurality of lens elements included in the multi-lens array may be additionally input to an image restoration model and may be used to determine a restored image.
Even if the lens elements are designed to have the same specification, the plurality of lens elements included in the multi-lens array may have different degradation characteristics due to process variations. Position information of the plurality of lens elements is additionally provided to the image restoration model, and thus, a restored image can be further determined based on a difference in degradation characteristics between the plurality of lens elements. Further, since a plurality of lens elements included in the multi-lens array are arranged in different positions, even if images are obtained by photographing the same scene, different levels of deterioration for the same subject in the images may occur. Position information of a plurality of lens elements is additionally provided to the image restoration model, and therefore, a restored image can be further determined based on the above degradation difference.
The above description of fig. 1A to 7 applies equally to the method of fig. 8, and therefore further description thereof is not repeated here.
Fig. 9 is a diagram illustrating an image restoration apparatus according to an example embodiment.
Referring to fig. 9, the image restoration apparatus 900 includes a memory 910, a processor 920, and an input/output (I/O) interface 930. The memory 910, processor 920, and I/O interface 930 may communicate with each other via a bus 940.
The processor 920 may be a device configured to execute instructions or programs, or to control the image restoration device 900, and may include, for example, a Central Processing Unit (CPU) and/or a Graphics Processor (GPU). The image restoration apparatus 900 may be connected to an external device (e.g., an image sensor or a database storing training data) via the I/O interface 930, and may exchange data. For example, the image restoration apparatus 900 may receive an input video through an external camera. Image restoration apparatus 900 may be implemented as part of any of a variety of computing devices, for example, a smart home appliance, such as a smart phone, wearable device, Personal Digital Assistant (PDA), tablet computer, laptop computer, or smart Television (TV), smart car, or kiosk. Further, the image restoration apparatus 900 may process the above-described operations.
According to example embodiments, when a convolution operation is performed in an image restoration model, information about a region of an image to be referred to may also be input to the image restoration model. Therefore, even for different levels of deterioration according to pixel positions, a variable operation can be applied on a region-by-region basis by a spatially varying image restoration function in a continuous space, so that a relatively high restoration performance is expected.
The example embodiments described herein may be implemented using hardware components, software components, or a combination thereof. The processing device may be implemented using one or more general purpose or special purpose computers, such as, for example, processors, controllers, and arithmetic-logic units, digital signal processors, microcomputers, field programmable arrays, programmable logic units, microprocessors, or any other device that is capable of responding to and executing instructions in a defined manner. The processing device may run an Operating System (OS) and one or more software applications running on the OS. The processing device may also access, store, manipulate, process, and create data in response to execution of the software. For the sake of brevity, the description of processing means is used in the singular; however, one skilled in the art will appreciate that the processing device may include a plurality of processing elements and a plurality of types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. Furthermore, different processing configurations are possible, such as parallel processors.
The software may include a computer program, code segments, instructions, or some combination thereof, to individually or collectively instruct or configure the processing device to operate as desired. Software and data may be embodied permanently or temporarily in any type of machine, component, physical or virtual device, computer storage medium or apparatus, or in a propagated signal wave that is capable of providing instructions or data to, or of being interpreted by, a processing apparatus. The software may also be distributed over network-coupled computer systems so that the software is stored and executed in a distributed fashion. The software and data may be stored by one or more non-transitory computer-readable recording media.
The method according to the above-described exemplary embodiments may be recorded in a non-transitory computer-readable medium including program instructions to implement various operations that can be performed by a computer. The media may also include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions recorded on the medium may be those specially designed and constructed for the purposes of the example embodiments, or they may be of the kind well known and available to those having skill in the computer software arts. Examples of non-transitory computer readable media include: magnetic media (such as hard disks, floppy disks, and magnetic tape), optical media (such as CD ROM disks and DVDs), magneto-optical media (such as optical disks), and hardware devices specially configured to store and execute program instructions (such as Read Only Memories (ROMs), Random Access Memories (RAMs), flash memories, etc.). Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described example embodiments, or vice versa.
While the present disclosure includes example embodiments, it will be apparent to those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims and their equivalents. The exemplary embodiments described herein are to be considered in all respects as illustrative and not restrictive. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and/or if components in the described systems, architectures, devices, or circuits are combined in a different manner and/or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is defined not by the detailed description but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents are to be construed as being included in the disclosure.
Claims (20)
1. An image restoration method comprising:
acquiring a target image; and
a restored image of the target image is obtained from the image restoration model based on the target image and the pixel position information of the target image.
2. The image restoration method according to claim 1, wherein the pixel position information includes an image representing a pixel position in the target image based on two-dimensional coordinates.
3. The image restoration method according to claim 1,
the pixel position information includes a first image in which the first value changes in a first axis direction in cartesian coordinates, and a second image in which the second value changes in a second axis direction perpendicular to the first axis direction, and
the first image and the second image have the same resolution as the target image.
4. The image restoration method according to claim 1,
the pixel position information includes a third image in which a third value changes based on a distance from a reference point in polar coordinates, and a fourth image in which a fourth value changes based on an angle with respect to the reference line, and
the third image and the fourth image have the same resolution as the target image.
5. The image restoration method according to claim 4, wherein the reference point is a principal point of a lens that captures the target image or a center point of the target image.
6. The image restoration method according to claim 1,
the pixel position information includes a first image in which a first value changes in a first axis direction in cartesian coordinates, a second image in which a second value changes in a second axis direction perpendicular to the first axis direction, a third image in which a third value changes based on a distance from a reference point in polar coordinates, and a fourth image in which a fourth value changes based on an angle with respect to the reference line, and
the first image, the second image, the third image and the fourth image have the same resolution as the target image.
7. The image restoration method according to claim 1, wherein the pixel position information and the target image are joined to each other and input to the image restoration model.
8. The image restoration method according to claim 1, wherein the pixel position information and the target image are input to a convolution layer of the image restoration model.
9. The image restoration method according to claim 1, wherein the target image includes different levels of degradation based on pixel position.
10. The image restoration method according to claim 9, wherein the degradation is caused by aberration of a lens used to capture the target image.
11. The image restoration method according to claim 1,
the target image includes at least one low resolution image having different levels of degradation based on pixel location, and
the restored image is a high resolution image with reduced degradation relative to the at least one low resolution image.
12. The image restoration method according to claim 1, wherein the image restoration model is trained to: a reference restored image having reduced degradation with respect to a reference target image is output in response to input of the reference target image having different levels of degradation based on the pixel position and reference pixel position information of the reference target image.
13. An image restoration method based on a trained neural network, comprising:
receiving a target image;
receiving pixel position information;
applying one or more transforms to the target image based on pixel location information in one or more layers of the trained neural network;
creating a restored image based on the one or more transforms applied to the target image; and
and outputting the restored image.
14. The image restoration method according to claim 13,
the step of receiving the target image includes: receiving a plurality of target images from a multi-lens array comprising a plurality of lens elements, an
The step of applying the one or more transforms comprises: the one or more transformations are also applied to the target image based on positional information of a lens element that captured each of the plurality of target images.
15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image restoration method of any one of claim 1 to claim 14.
16. An image restoration apparatus comprising:
a memory storing one or more instructions; and
at least one processor configured to execute the one or more instructions to:
receiving a target image;
receiving pixel position information;
applying one or more transforms to the target image based on pixel location information in one or more layers of the trained neural network;
creating a restored image based on the one or more transforms applied to the target image; and outputting the restored image.
17. The image restoration device according to claim 16, wherein the pixel position information includes an image representing a pixel position in the target image based on two-dimensional coordinates.
18. The image restoration device according to claim 16,
the pixel position information includes a first image in which the first value changes in a first axis direction in cartesian coordinates, and a second image in which the second value changes in a second axis direction perpendicular to the first axis direction, and
the first image and the second image have the same resolution as the target image.
19. The image restoration device according to claim 16,
the pixel position information includes a third image in which a third value changes based on a distance from a reference point in polar coordinates, and a fourth image in which a fourth value changes based on an angle with respect to the reference line, and
the third image and the fourth image have the same resolution as the target image.
20. The image restoration device according to claim 16, wherein the pixel position information and the target image are linked to each other.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020190155506A KR20210066366A (en) | 2019-11-28 | 2019-11-28 | Method and device for restoring image |
| KR10-2019-0155506 | 2019-11-28 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112862695A true CN112862695A (en) | 2021-05-28 |
Family
ID=71143530
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010362078.3A Pending CN112862695A (en) | 2019-11-28 | 2020-04-30 | Image restoration method and apparatus |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US11941781B2 (en) |
| EP (1) | EP3828813A1 (en) |
| KR (1) | KR20210066366A (en) |
| CN (1) | CN112862695A (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200137380A1 (en) * | 2018-10-31 | 2020-04-30 | Intel Corporation | Multi-plane display image synthesis mechanism |
| KR20220080249A (en) | 2020-12-07 | 2022-06-14 | 삼성전자주식회사 | Method and apparatus for processing restore image |
| CN114331912B (en) * | 2022-01-06 | 2023-09-29 | 北京字跳网络技术有限公司 | Image restoration method and device |
| CN114549369B (en) * | 2022-04-24 | 2022-07-12 | 腾讯科技（深圳）有限公司 | Data restoration method and device, computer and readable storage medium |
Family Cites Families (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3444612B2 (en) | 1992-12-22 | 2003-09-08 | 富士通株式会社 | Color correction method of image input device by neural network |
| US7218751B2 (en) * | 2001-06-29 | 2007-05-15 | Digimarc Corporation | Generating super resolution digital images |
| US7545422B2 (en) * | 2004-10-27 | 2009-06-09 | Aptina Imaging Corporation | Imaging system |
| JP4764624B2 (en) * | 2004-12-07 | 2011-09-07 | 株式会社 日立ディスプレイズ | Stereoscopic display device and stereoscopic image generation method |
| US8254713B2 (en) * | 2005-11-11 | 2012-08-28 | Sony Corporation | Image processing apparatus, image processing method, program therefor, and recording medium in which the program is recorded |
| US8558899B2 (en) * | 2009-11-16 | 2013-10-15 | The Aerospace Corporation | System and method for super-resolution digital time delay and integrate (TDI) image processing |
| US8878950B2 (en) * | 2010-12-14 | 2014-11-04 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using super-resolution processes |
| CN110780707A (en) | 2014-05-22 | 2020-02-11 | 索尼公司 | Information processing apparatus, information processing method, and computer readable medium |
| EP3182372B1 (en) * | 2015-12-18 | 2019-07-31 | InterDigital CE Patent Holdings | Method and system for estimating the position of a projection of a chief ray on a sensor of a light-field acquisition device |
| US10311326B2 (en) | 2017-03-31 | 2019-06-04 | Qualcomm Incorporated | Systems and methods for improved image textures |
| US11257856B2 (en) * | 2017-10-13 | 2022-02-22 | Trustees Of Boston University | Lens-free compound eye cameras based on angle-sensitive meta-surfaces |
| CN107679580B (en) | 2017-10-21 | 2020-12-01 | 桂林电子科技大学 | Heterogeneous migration image emotion polarity analysis method based on multi-mode depth potential correlation |
| JP2019087778A (en) | 2017-11-01 | 2019-06-06 | 日本放送協会 | Image restoration filter and learning apparatus |
| US11030724B2 (en) | 2018-09-13 | 2021-06-08 | Samsung Electronics Co., Ltd. | Method and apparatus for restoring image |
| US11375092B2 (en) * | 2018-10-04 | 2022-06-28 | Samsung Electronics Co., Ltd. | Image sensor and image sensing method |
| KR20200072136A (en) * | 2018-12-12 | 2020-06-22 | 삼성전자주식회사 | Lens array camera and method of driving lens array camera |
-
2019
- 2019-11-28 KR KR1020190155506A patent/KR20210066366A/en not_active Application Discontinuation
-
2020
- 2020-04-16 US US16/850,724 patent/US11941781B2/en active Active
- 2020-04-30 CN CN202010362078.3A patent/CN112862695A/en active Pending
- 2020-06-24 EP EP20181976.0A patent/EP3828813A1/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| US11941781B2 (en) | 2024-03-26 |
| EP3828813A1 (en) | 2021-06-02 |
| US20210166352A1 (en) | 2021-06-03 |
| KR20210066366A (en) | 2021-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11941781B2 (en) | Method and apparatus for restoring image | |
| CN108111714B (en) | Multi-lens based capture apparatus and method | |
| US11663691B2 (en) | Method and apparatus for restoring image | |
| US10977831B2 (en) | Camera calibration method and apparatus based on deep learning | |
| US11663699B2 (en) | Method and apparatus for restoring image | |
| CN113168670A (en) | Bright spot removal using neural networks | |
| US11375092B2 (en) | Image sensor and image sensing method | |
| US10924667B2 (en) | Image sensor and image sensing method | |
| WO2022100490A1 (en) | Methods and systems for deblurring blurry images | |
| US11734877B2 (en) | Method and device for restoring image obtained from array camera | |
| CN113645422A (en) | Image sensor with a plurality of pixels | |
| KR102614908B1 (en) | Device and method to restore image | |
| KR20200072136A (en) | Lens array camera and method of driving lens array camera | |
| US10868957B2 (en) | Apparatus and method for processing image to reconstruct image | |
| KR20220080249A (en) | Method and apparatus for processing restore image | |
| Que et al. | Lightweight and Dynamic Deblurring for IoT-Enabled Smart Cameras | |
| US11651475B2 (en) | Image restoration method and device | |
| Alhawwary | Depth and IMU aided image deblurring based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |