LT6525B - Method for the enhancement of digital image resolution by applying a unique processing of partially overlaping low resolution images - Google Patents

Abstract

The object of the present invention is a method for the composition of high-resolution image performing automated enhancement of image resolution from several partially overlapping low-resolution images captured at different positions and angles of photo cameras Formation of the enhanced resolution image includes the 3 main stages: in the first stage a 3D model of the surface captured in photographs is identified: positions and turning angles of cameras are restored, as well as the 3D surface of the object captured in photographs is calculated; in the second stage a correction of optical distortions of all lower-resolution images is performed, the required amount of correction is determined using the geometric distortion coefficients received in the first phase; in the third stage, the calculation of the intensities of enhanced resolution pixels is carried out - to this end, for each new enhanced resolution point a 3D point located on a 3D mesh of the captured object is found, when projecting the 3D point to all lower-resolution points, the projections of the 3D point are found, a four-square area of the point environment, the angles of which are also projected to all lower-resolution images, is marked around the 3D point, from the lower-resolution points located in the projections of the 3D point environments an equation system is formed, by resolving which the wanted intensity of the enhanced resolution point is found.This paper describes the principles for the enhancement of the monochrome image resolution, but it can also be used for colour images. In the case of colour images the colour of each enhanced resolution pixel is determined by adapting the described method to each original colour (red, green and blue) individually.

Description

TECHNICAL FIELD

The invention relates to the field of electronic image (photo) processing, and more particularly to increasing the resolution (resolution) of electronic images by utilizing the data of several overlapping low resolution images.

TECHNICAL LEVEL

The present invention provides the ability to automatically create a high resolution image from multiple overlapping low resolution images captured at different camera positions and angles. In the process, the positions of the cameras are reconstructed from the initial low-resolution images and the spatial surface used to connect the low-resolution images. After geometric correction of the initial low-resolution images, the intensities of the high-resolution pixels are calculated from the projections of the spatial surface points they represent on the low-resolution images.

The following state-of-the-art is discussed below, i.e. information that is currently publicly available.

Known Patent US8582922B2 (published November 12, 2013), which discloses a method of increasing the resolution of an image using two or more low resolution images. The resolution enhancement method is based on the calculation of values of high resolution pixels from low resolution pixels using the principle of weighted interpolation. This method of calculation does not mention anything that has been captured at different angles of rotation of the camera to the subject, without increasing the resolution of the 3D surface, which may result in using images of a single plane or planes with only a few angles of degree parts.

Also known is US20040170340A1 (published September 2, 2004), which discloses a method for increasing the resolution of an image, wherein the high resolution image is obtained from at least two lower resolution images using the Bayesian method. However, this method of resolution enhancement can only be used when the orientation of the initial low-resolution images and the rotation of the subject are very close.

Also known as US20030012457 (published January 16, 2003). It produces a higher resolution flat image of several low resolution flat images. The basic principle of this method is that all the information is gathered in one image and then, according to the appropriate model, clipped to unnecessary details.

In most cases, prior art documents are generally associated with image resolution enhancement techniques that focus on composing low-resolution multiple images and further processing them using a variety of techniques. Also, documents usually deal with images reproduced in a two-dimensional plane that coincides with an image plane. As image processing and shaping technology has only begun to develop more rapidly in the last decade (especially in recent years) as a field of science and application, it can be said that image processing technologies are currently experiencing a breakthrough. This is due to the fact that current computer technology is able to process large amounts of information quickly without too much simplification of the original images, i.e., the ability to work with images using a variety of sophisticated methodologies for better results.

The vast majority of current resolution techniques in the state of the art rely on the reconstruction of small changes in camera position (displacement and rotation). Position search is performed in 2D space coinciding with the image plane. When the optimal positions are found, the increased resolution is calculated for the entire area of the image being reproduced at one time. However, with large changes in camera position and capture direction, homography-based methods do not work because images can no longer overlap due to perspective transformation of objects captured at different angles.

The method of the present invention performs image alignment during a photogrammetry process. Key-points and 3D model Bundle Adjustment find the relationships between images and camera positions in 3D. Due to the different photo capture position, captured objects are visible in different areas of each image overlay, i.e., it is not possible to increase the resolution of the entire image at once because images cannot overlap even after the required transformations have been made. This problem is solved by calculating each pixel at a higher resolution. The sequence of steps used in the present invention reproduces all of the higher resolution points by locating each corresponding 3D space point on the 3D surface, marking it a local surface plane, and projecting this small square into all low resolution images capturing this 3D surface location. Based on the projections obtained, each sample selected contains the intensities of the projection points, which are then used to construct the system of equations. After solving the system of equations, the obtained central intensity of the studied point is preserved in an enlarged resolution image.

THE SUBSTANCE OF THE INVENTION

During the 3D image conversion process, the 3D surface of the captured object (ground, buildings, canopy, etc.) is reproduced from the original low-resolution images and used to find projections of the images captured in the photographs. Higher resolution image reconstruction is calculated using overlapping low resolution photo areas. Higher resolution pictures are obtained after all processing steps. These photos correspond to the position and angle of capture of the original photos (i.e. they show the same images), but have a higher image resolution so the image captured on them becomes more detailed. The image will be more detailed in areas where larger numbers of photos overlap.

There are three basic steps to creating a higher resolution image:

In the first stage, using a photogrammetry method, a 3D model of the surface captured in the photos is found: the positions and angles of the cameras are restored, and the 3D surface of the object captured in the photos (mesh) is calculated.

The second stage involves optical lens correction (radial undistortion) for all low-resolution images. The amount of correction needed is determined using the geometric distortion coefficients obtained during the photogrammetry.

The third step involves calculating the intensities of the higher resolution pixels. For this purpose, a new 3D point is found on each new higher resolution point on the 3D surface of the captured object. Projecting a 3D point to all low-resolution images results in spot projections. Around the 3D point is a rectangular area of the point environment, the angles of which are also projected onto all low-resolution images. For further calculations, only low-resolution images with a well-defined point of search are selected. The low-resolution points in the projections of 3D point environments make up a system of equations that solves the search for the desired high-resolution point intensity.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1. Overlapping points of (a) low-resolution, (b) high-resolution, (c) magnified and low-resolution environments of the target point environment.

Fig. 2. The location of the searched point in different low resolution images.

PREFERRED EMBODIMENTS

As used herein, the term "resolution" or "resolution" refers to a parameter of an image reproduced by electronic equipment that describes the number of pixels per unit length / area / space.

As used herein, the term "image point" or "point" refers to the square / cubic area of an image that is occupied by that point.

As used herein, the term "pixel intensity" or "intensity" is understood to mean the luminance of a pixel (for monochrome images) or the luminosity of the pixel color (red, green, and blue) (for color images).

In this description, the term "low resolution image" or "lower resolution image" refers to a photograph of an object that clearly shows the captured object but depicts small details of the object in the abstract.

As used in this description, the term "high resolution image" or "high resolution image" refers to an enhanced photograph of the same subject that details small details of the subject (for example, a note could not be read in the original low resolution image).

In the present invention, image rendering and processing can be performed on electronic image-capable devices such as a computer, a camcorder, a VCR, or a video display device (e.g., a television, monitor). The presented image processing requires electronic devices capable of processing digital information (various memory modules, at least one processor, input, output devices, etc.).

High resolution images capture the captured object in more detail and can be used to solve more complex tasks. However, high-resolution shooting is limited by a variety of reasons: technical (the optics of the imaging device prevent zooming in), legal (the resolution of satellite and thermal imaging is limited by law because of their strategic importance), etc. . One part of the methods increases the resolution of the image using one low-resolution image, while the other methods do so using two or more low-resolution images. In the first case, the resolution of the image is increased by various means of interpolation, and in the second, low-resolution images are mapped according to the found focal points and algorithmically attempts to detect small changes in the camera position and their compensation. Other methods such as noise cancellation, brightness enhancement, and more can be used in such methods.

The resolution techniques discussed above treat the captured image as a plane, requiring the initial low-resolution images to be captured from a single point (constant camera position). The method described in this description for increasing the resolution uses initial low-resolution images captured from different camera positions and at different tilt angles to the subject being captured. In the present invention, the formation of the high resolution image consists of three steps of increasing the resolution.

In the first step of resolution resolution, captured photographs reproduce the 3D surface of the subject. The 3D surface is reproduced using photogrammetry, determining the camera position and angles of rotation with respect to the subject being captured.

In the second stage, optical distortion is corrected using geometric distortion values obtained during photogrammetry.

The third step is to determine the intensities of the pixels in the high resolution image.

The present invention is novel in that the three steps of image resolution enhancement produce high resolution images where the initial low resolution images were captured at different camera positions and tilt angles. The method described works best if the beams are output at each point on the surface and the position of the cameras that see these points is more than 5 degrees.

The steps to increase the video resolution are described in more detail below:

The resolution is increased from N low-resolution images that form a set of low-resolution images Z = {K ₀ , F _{] 5} K, V _n }. Each of the original low-resolution images V, changes to the high-resolution image Vj. As you change the resolution of the image, the geometry of the image remains unchanged and the number of pixels that make up the image increases. Resolution increase is done at any size, but not more than 1.5 times is recommended, because incrementing more than 1.5 times may result in unstable results when calculating resolution.

Fig. Figure 1 illustrates a possible 1.5x magnification: (a) the image shown in Part 1 has two low-resolution pixels; (b) the part of the image represented by the part shall consist of three dots of enlarged resolution; (c) The section shows the overlap between low resolution and higher resolution images.

During each step of resolution resolution, each pixel of the magnified image is processed separately, i.e., the entire sequence of functional treatments is applied to each pixel of the magnified image being formed.

The individual phases (steps) of the functional processing sequence are described in more detail below:

1. A radius R 'passing through the center C of the lower resolution image V intersects the desired higher resolution pixel Q' _xy . At the intersection of the radius with the reconstructed 3D surface, the point X is marked.

2. Auxiliary plane Z leads through point X. The normal direction of plane Z is opposite to the direction of radius R ,.

3. A quadratic environment ^ is created around point Q _xy in the high-resolution image. 5. The square point βζ, environment 5, consists of high-resolution pixels e S. The number of pixels, that is, the size of the environment, is freely chosen. Fig. 1, b) shows a square dot environment with three high resolution pixels.

4. The radii are passed through adjacent points wj _v e 5 of Q _xy , whose three-dimensional points T _uv are intersected at the intersection plane Z.

5. At this stage, the two-dimensional image is aligned with the three-dimensional image. Auxiliary Z plane is shifted, rotated so that its normal coincides with the three-dimensional surface normal N at the point X. Points _x T _uv rotated together with plane Z

6. In each lower resolution image V _k eL, using three-dimensional points T _uv , projected by photogrammetric projection matrices, their projections W ^k v are obtained. Because of the tilt of the plane Z relative to the three-dimensional surface, the point projections W _u ^k v in each image will be rendered by irregular quadrangles that best correspond to the projection of the three-dimensional surface area in each image.

7. A new image set K is formed from the former image set L and the image is transferred from the set L to the new image set K if two conditions are met:

a) All projections W ^k v fall on the image V _k ie: VW ^k v: Q <u <-w _Vt , Q <v <h _Vt , where w _{V / in} and h are the width and height of the image V _k respectively.

b) The normal N _x of the point X has a smaller angle with the radius R _k of the image V _k

Γ · Ί than 60 'ty arccos <y. Here n is the normal N _{x of} the point X written in vector

Vii ⁿ llll ^r liy form, or - the radius -R _{k of} the image V _k is inscribed in vector form.

8. For all lower resolution images, the intensities G ^k v are calculated at points W ^k v of the projections V _k e K. For the determination of intensities, bicubic interpolation is used.

9. Based on the description in the section entitled "Formation of the system of equations" and the obtained values of low-resolution image intensities G ^k m, a system of normal equations for calculating the intensities of the pixels G ' _w is created.

10. After solving the system of equations, select the middle element x _t : i = M / 2 from the vector x corresponding to the intensity G ' _{m of} the central point of magnification _, and write it to point Q' _{xy in} magnification image V).

11. The processing mechanism is repeated for the remaining V points of the image.

Fig. Figure 2 shows the environment of the target point in different low-resolution images.

The following explanation of the above system of equations is given below:

Each pixel of an image is understood as the average intensity of the square surface area occupied by that pixel. Lower-resolution pixels overlap non-ideally, resulting in a few high-resolution pixels (some of them partially overlapping) in the quadrilateral of each pixel. In general, the value of each lower resolution point can be calculated using a weighted average formula:

m a = l v = l (1)

Here is the intensity of the native pixel, the size of the S-edge of the m-pixel environment (Fig. 1 shows an example with the edge equal to 3), g ' _uv - the intensity of the pixel at high resolution, w' is calculated as the ratio of g ' _uv overlapping area to g _y .

The value of the high-resolution point gius is calculated from the projection of this point's environment S on each low-resolution image. Fig. Figure 1 shows the point environment in the low resolution image (part a), the high resolution image (part b), and the overlapping areas of the high and low resolution pixels (part c).

Using the formula (1), a system of linear equations (eguation system) is created and written in matrix form:

Ax = b

Here, x is the intensities of all the high-resolution pixels you are looking for, b is the intensity of the initial low-resolution pixels, A is the matrix of weights corresponding to the overlapping area of each high-resolution pixel with the low-resolution pixels.

Given the number of high-resolution points (unknowns) entering the point environment, the required number of equations in the system is determined. The number of equations must be the same or greater than the number of searched points (unknowns). Fig. In the example 1, the high-resolution point environment consists of 9 unknown points, so the system of equations must contain at least 9 equations. The low-resolution environment consists of 4 dots. To achieve the required number of equations, we need to use data from at least 3 low-resolution images to form a system of 12 equations, each with 9 unknowns.

Referring to FIG. 1, and the available values of low-resolution pixel intensities G * _{v of the} target pixel, provide a system of normal equations for calculating the intensities of the high-resolution pixels G ' _w . The sticker

A is filled by the ratio of the areas of the overlapping points. Each equation corresponds to an estimate of the intensity of one low-resolution point consisting of high-resolution estimates of unknown points according to formula (1). The vector b is filled with low-resolution point intensities G ^. In the example shown, the first 4 equations describe the environment points of the first low-resolution image, the following 4 for the second image, and the last 4 for the third image:

A =

4/9 2/9 0 2/9 1/9 0 0 0 0 ' 'θ \' 0 2/9 4/9 0 1/9 2/9 0 0 0 Gh 0 0 0 2/9 1/9 0 4/9 2/9 0 G \ 0 0 0 0 1/9 2/9 0 2/9 4/9 g \ ₂ 4/9 2/9 0 2/9 1/9 0 0 0 0 Gh 0 2/9 4/9 0 1/9 2/9 0 0 0 , b = Gi 0 0 0 2/9 1/9 0 4/9 2/9 0 G ^ 0 0 0 0 1/9 2/9 0 2/9 4/9 g ² 22 4/9 2/9 0 2/9 1/9 0 0 0 0 G ^ 0 2/9 4/9 0 1/9 2/9 0 0 0 tf ₂ 0 0 0 2/9 1/9 0 4/9 2/9 0 GL _ 0 0 0 0 1/9 2/9 0 2/9 4/9 _1

After solving the system of equations, the intensities G ' _{uv in the} vector x.

find the points of increased resolution _ Γζ »'zV sif s-n z-»' z ~ »» z - »/ V - ^ 11 ^ 12 ^ 13 ^ 215 ^ 22 ^ 23 ^ 315 ^ 32 ^ 33 J

Matrix A will be the same for all calculated points (or can be extended if more than 3 images are used in the calculation), and vector b will be filled with the values of the surrounding point of the searched point.

The following are limitations of the resolution enhancement mechanism.

The resolution will be different for different areas of the high resolution image you have created. The detail of a high resolution image will depend greatly on both the amount of overlapping photos and the total pixel density in the target spot environment, ie: in closer shots, the subject's surface has a higher resolution than in longer shots; overlapping pictures taken at a greater distance will result in unstable results. If the coordinates of the overlapping pixels are highly correlated, the pictures may not be sufficiently complementary as there will be no new information that can be used to increase the resolution of the reproduced / formatted picture. However, due to the unevenness of the surface being studied and changes in the camera's position elsewhere in the same image, the correlation of the points will be lower, resulting in a higher resolution image.

In areas of the picture where abstract objects (sky, glossy surface, patchy water, etc.) are captured or there is not enough overlap, the resolution is enhanced from the initial low-resolution pixels using bicubic interpolation.

Although this work describes the principles of increasing the resolution of monochrome images, this method can also be used for color images. For color images, the color of each pixel in the enlarged resolution is determined by applying the method described for each primary color (red, green, and blue).

In order to illustrate and describe the present invention, the following description of preferred embodiments is provided. It is not a detailed or restrictive description to determine the exact form or embodiment. the description above should be viewed as an illustration rather than a limitation. It will be appreciated that many modifications and variations may be apparent to those skilled in the art, and an embodiment has been selected and described to best explain the principles of the invention and their best practical application for different embodiments with different modifications suitable for a particular application or embodiment. for customization. The scope of the invention is intended to be defined by the appended definition and its equivalents, in which all the foregoing terms have the widest possible meaning, unless otherwise stated.

Embodiments described by those skilled in the art may be made without departing from the scope of the present invention as defined in the following definition.

Claims (3)

DEFINITION OF INVENTION 1. A method of automatically forming and processing overlapping low-resolution images taken at different angles and angles of the camera to produce a high-resolution image comprising: collecting information from low resolution images; processing mechanism; and high resolution image forming; characterized in that this processing method comprises the following three functional processing steps: in the first step of increasing the resolution from low resolution photos using photogrammetry techniques, finding camera positions and angles of rotation in space, and reproducing the 3D surface of the captured subject; in the second step of increasing the resolution, optical distortion is corrected by using geometric distortion data obtained during photogrammetry; in the third step, the values of the intensities of the points constituting the enlarged resolution image are calculated. 2. A method of automatically forming and processing digital images according to claim 1, characterized in that said 3 processing steps comprise the following functional steps: a) passing through the center C of the low-resolution image V a radius R ,, intersecting the desired high-resolution image point Q _xy , where X is the intersection of the beam with the reconstructed three-dimensional surface; b) passing through the point X an auxiliary plane Z, where the normal direction of the plane Z is opposite to the radius R; c) creating a square point βζ around the point Q ^l xy in the magnified image, whereby the square point ^ is surrounded by high-resolution pixels e S, where: the size of the pixel is optional; d) radii passing through adjacent points W ' _v e S of Q ^l xy, whose three-dimensional points T _uv are intersected at the intersection plane Z; (e) Perform a two-dimensional image matching procedure with three-dimensional image, where the auxiliary plane Z is tilted, rotated so that its normal N _z coincides with the three-dimensional surface normal N _x at X, and the points T _uv are rotated to remain in the same plane Z in places; f) at each lower resolution image V _k E l using photogrammetry results designed three-dimensional points T _uv, resulting in the projections of the W ^k v, where a plane of the Z tilt three-dimensional surface and the projection W ^k v in each image will be displayed with an irregular quadrangle which best matches the projection of the 3D surface area in each image; (g) a new image set K is formed from the former image set L, and the image is transferred from the set L to a new image set K if two conditions are met: (aa) all projections W ^k v fall on the image V _k ie: VfF ^k : 0 <M <w _rt , 0 <v <≤ _t , where w and h _Vt are the width and height of the image V _k respectively; bb) the normal line at X N _x V _k video _picture is a radius R smaller than the angle of 60 "or arccos _(r \ No <y, where n - point X X _X normal line vector written form or visual - V _k radius R _{k is} in vector form; (h) For all lower-resolution images, the values of intensities G ^k v shall be calculated at points W ^k v of the projections V _k e K, whereby a bicubic interpolation technique shall be used to determine the intensities; (i) deriving the lower-resolution image intensities G ^k v to produce a system of normal equations for the calculation of the higher-resolution image intensities G '_m; j) solving the above system of equations, selecting from the vector x a middle element x,: i = M / 2 corresponding to the intensity of the central magnification point G ' _uv and writing the j point Q' _{xy in the} magnification image Z /. 3. A method of forming and automatically processing digital images according to claim 2, wherein said functional steps a) to j) are repeated for the remaining pixels of the image.