RU2654127C1 - Method for generating a digital panoramic image - Google Patents

Abstract

FIELD: image forming devices.

SUBSTANCE: invention relates to computer technology, namely to the means for generating a high-resolution digital panoramic image from several images. Method of forming a digital panoramic image from several sources: images from cameras located on the faces of a regular polyhedron are received and processed; each image is cut along the boundaries of a section of the regular polyhedron; composite spherical panoramic image is formed; image correction is made on the model of the field vector diagram with distortion correction; images are cropped; in the overlapping areas of images using the SURF method of a pair of singular points is calculated and additionally, on the basis of affine transformations, a nonlinear deformation of each cropped image is performed from data on pairs of singular points in the overlapping areas of images; address matrices of the location and the accessories of an individual pixel to the faces of the regular polyhedron in the Cartesian coordinate system are calculated; projection of the regular polyhedron onto a sphere inscribed into it is constructed; image stitching is performed.

EFFECT: technical result is to improve the quality of the high-resolution digital panoramic image.

1 cl, 12 dwg

Description

The invention relates to computer technology and can be used to generate a high-resolution digital panoramic image from several images obtained using a multi-camera all-round viewing device for providing surveillance video surveillance of the safety of special, industrial, administrative and other objects.

Terms within this application

A sensor is a photosensitive element of a digital camera, which is a matrix of millions of miniature pixel cells.

ARM (Advanced RISC Machine) is a microprocessor architecture with a reduced set of instructions.

MiPi (Eng. Mobile Industry Processor Interface - processor interface for mobile systems) is a standard interface for exchanging data between a digital camera sensor and a computing node.

FPGA (Field-Programmable Gate Array - a user-programmable gate array) - a semiconductor device that can be configured by the manufacturer or developer after manufacturing, is also called FPGA.

RAW (Eng. Raw - raw, raw) - a digital photo format containing raw data obtained from a photosensitive sensor.

RGB (English Red, Green, Blue - red, green, blue) is an additive color model that describes the method of color synthesis for color reproduction.

SURF (English speeded up robust features) is a method that simultaneously searches for special points in the image and builds their description, which is invariant to zooming and rotation. [Wow N., Ess A., Tuvtelaars T., Gool L. V. Speeded-Up Robust Features (SURF) // Proceedings of the 9th European Conference on Computer Vision. Springer LNCS. 200G. Vol. 3951. Pt. 1. P. 404-417].

A special point is the point of maximum change in the brightness gradient in a certain neighborhood in the image at which the determinant of the Hessian matrix reaches an extremum.

Affine transformations are a common method for image deformation, which allows you to rotate, stretch and reflect the image in one operation.

Stitching - creating frames based on the usual format with a small viewing angle of a single panoramic image.

A known method of obtaining panoramic images and a device that implements it (US patent No. 5563650, H04N 7/18, from 08.10.1996). The method is based on obtaining images using a camera with a fish eye lens (lens with an ultra-wide viewing angle), subsequent conversion of images into digital form and transformation into panoramic images devoid of distortion visible to the human eye using a computer and special software .

The disadvantages of this method is the possibility of applying this method only to the claimed device - a camera with a lens type "fish eye", respectively, a narrow scope, as well as the need to obtain a large number of images to ensure high accuracy of obtaining panoramic images.

A known method of creating panoramic images in digital camera systems (US patent 6714249, H04N 7/00; G03B 17/00, 03/30/2004), in which the camera is prepared and installed in a certain position in advance, turning it to a predetermined position, several series are obtained images with areas intersecting at the edges, store these images in memory and automatically combine several images into a panoramic image.

The disadvantages of this method is the obtaining of panoramic images with low accuracy due to the lack of correction of aberrations of optical systems, namely radial distortion, as well as the difficulty of practical implementation of the method in connection with the installation of the camera in a strictly defined position and its rotation in a predetermined position.

Closest to the proposed method is a method implemented by a device for a dodecahedral imaging system (US patent US5023725, H04N5 / 74; H04N7 / 00; H04N7 / 18; H04N9 / 31, from 11.06.1991), in which a spherical image is obtained as a result of processing images obtained from 12 cameras located on the faces of the dodecahedron. Moreover, each of the faces of the dodecahedron forms a pentagonal section in the form of a section of a concentric sphere. The image received from each camera is cropped to the borders of its pentagonal section by masking the signal from the camera. At the stage of forming a composite panoramic spherical image, electro-optical correction of these images is performed, and the resulting panoramic image is presented in the form of a sphere.

The disadvantages of this method are due to the use of electro-optical correction of aberrations of optical systems, which does not allow to provide high visual quality of a panoramic image while eliminating the effects associated with the distortion of images received from cameras, and also does not take into account the displacement of one image relative to another.

The task of the invention is to expand the arsenal of methods for generating a digital panoramic spherical image based on images obtained from several cameras.

The technical result of the use of the present invention is the high visual quality of a high-resolution digital panoramic image due to the alignment of geometric distortions associated with the distortion of the images received from the cameras and eliminating the discrepancy in the sizes and shapes of images of objects located in the overlapping area of adjacent images obtained from each camera.

The task is achieved by the proposed method for generating a digital panoramic image from several sources, including processing images obtained from cameras located on the edges of a regular polyhedron by cropping images under the section boundaries in the form of a regular polygon, forming a composite spherical panoramic image with correction of the received images, into which The following new features:

- before cropping the images, they are corrected based on the model of the vector field diagram by compensating for distortions caused by the lens of each digital camera, taking into account the distortion coefficient and image scaling factor;

- cropping the images obtained from each camera is performed by determining the ownership of each pixel of the input image to the correct n-gon inscribed in it, where n is from 3 to 5;

- additionally, on the basis of affine transformations, a nonlinear deformation of the cropped image is performed according to pairs of singular points in the areas of overlapping images coming from neighboring digital cameras. Moreover, pairs of singular points are determined using the SURF method;

- for the formation of a composite panoramic image, address matrices are preliminarily calculated that determine the location and belonging of an individual pixel of each image to the faces of a regular polyhedron, for example, a dodecahedron, hexahedron, icosahedron, octahedron or tetrahedron, in a Cartesian coordinate system, then — constructing the projection of the panoramic image onto the inscribed sphere into a regular polyhedron, using FPGA and converting Cartesian coordinates X, Y, Z to spherical, and then using software eniya for constructing equidistant cylindrical projection sphere carried crosslinking images.

The proposed method meets the conditions of “novelty” and “inventive step” due to the fact that it allows maximum elimination of undesirable effects of geometric distortions from the mismatch of the sizes and shapes of images of objects located in the overlapping area of adjacent images at the boundaries of the alignment of n - coal sections due to:

- nonlinear deformation of images constituting a panoramic image, produced from data on pairs of singular points detected in areas of overlapping images coming from neighboring digital cameras;

- the use of pre-calculated address matrices based on the model of the regular polyhedron in the formation of a composite panoramic image, which allows you to determine the location and belonging of a single pixel of each image to the faces of the regular polyhedron in the Cartesian coordinate system.

The invention is characterized by the following figures.

FIG. 1 - Development of a dodecahedron model on a plane, taking into account the location of its pentagonal sections.

FIG. 2 - The location of the coordinate system for the area of the photosensitive sensor A of the digital camera and the numbering of the sides and angles of the pentagonal section B, where Roman numerals denote the sides, Arabic numerals indicate the angles of the pentagonal section.

FIG. 3 - Model of a dodecahedron with numbers of sides and faces forming its equilateral pentagonal sections: a) front view; b) rear view.

FIG. 4 - Development plan of the cylindrical projection of the dodecahedron onto the plane.

FIG. 5 - The spatial coordinate system of the dodecahedron and the sphere inscribed in it.

FIG. 6 - The result of constructing an equidistant cylindrical projection of a sphere onto a plane.

FIG. 7 - Correction of distortions of the optical system of a digital camera on the example of the image from the camera sensor 9: a) image from the sensor 9 without correction; b) image from sensor 9 after correction.

FIG. 8 - Cropping of the corrected image from the camera 9 sensor to the borders of its pentagonal section: a) image from the camera 9 sensor with the pentagonal section template marked on it; b) the result of cropping the image obtained from the camera sensor 9.

FIG. 9 - Areas of overlapping cropped images coming from sensors of adjacent digital cameras and the result of matching found special points marked with “*”, where a) cropped image from camera sensor 5; b) cropped image from the camera sensor 9.

FIG. 10 - The result of combining the boundaries of the images obtained from adjacent sensors of cameras 5 and 9, according to pairs of singular points.

FIG. 11 - A fragment of the projection of a spherical image onto a plane for images obtained from adjacent sensors of cameras 5 and 9.

FIG. 12 - The result of stitching a composite panoramic image on a plane.

The proposed method for the case when the device contains 12 digital cameras located on the edges of the dodecahedron, is implemented as follows:

1. For image processing, images from 12 digital cameras located on the edges of the dodecahedron are obtained using a single sync signal. Each camera consists of a lens and a photosensitive sensor. All images via the MiPi interface are transferred to the memory of their own computing node, implemented on the basis of the FPGA and ARM processor, for preliminary image processing;

2. Convert from RAW to RGB format for each image. This conversion is necessary for more convenient presentation of images and their further processing. The conversion result is written to the intermediate buffer;

3. Before cropping the images, the distortions caused by the digital camera lens are corrected taking into account the distortion coefficient and scaling factor. Correction is made on the basis of a vector field diagram model (Gelman R.N., Dunts A.L. Laboratory calibration of digital cameras with great distortion // Geodesy and Cartography. M., 2002. No. 7, p.23-31);

4. Using FPGA, each image obtained from the sensor (A) is cut off under the boundaries of its pentagonal section (B) by determining whether each pixel of the input image belongs to the pentagon inscribed in it, as shown in FIG. 2, where the Roman numerals are encoded sides, and in Arabic - the corners of the pentagonal section in accordance with the coordinate system (x, y);

5. Using the means of the ARM processor, in the areas of overlapping images from neighboring sensors, the coordinates of the singular points are calculated based on the SURF method, which simultaneously searches for singular points and constructs their description as a vector of values, which is invariant to zooming and rotation;

6. Compare pairs of singular points having the same description on the images of adjacent sensors;

7. Perform non-linear deformation of each image, according to pairs of singular points, based on affine transformations;

8. The address matrices are calculated that determine the location and belonging of a single pixel of each image to the faces of the dodecahedron in the Cartesian coordinate system, based on the mathematical model of the dodecahedron, graphically presented in figure 3, where the coding system of the sides and angles of the dodecahedron is used in accordance with the system presented in figure 2.

Figure 4 shows a representation of a panoramic image when projecting a dodecahedron model onto a plane;

9. To obtain a projection of the panoramic image onto the sphere inscribed in the dodecahedron, using Cartesian coordinates, the Cartesian coordinates X, Y, Z are converted into spherical, forming a spherical composite panoramic image as shown in FIG. 5, where the cube is three-dimensional space, r is the radius of the sphere inscribed in the dodecahedron.

10. Stitching is carried out, forming a composite panoramic image using software for constructing an equidistant cylindrical projection of a sphere, as shown in FIG. 6, where the numbers indicate the numbers of the sensors. This method of presenting a panoramic image allows you to observe the whole picture as a whole and is more convenient for display using standard graphic devices such as a TV, monitor, projector.

An example implementation of the method.

A panoramic camera containing 12 digital cameras located on the edges of the dodecahedron was installed in the center of the room.

After turning on the camera, the connected systems were initialized and the necessary settings were made.

By pressing the “record” button from each of the 12 sensors of digital cameras, the resulting image is transmitted to its own computing node, where conversion from RAW format to RGB format is performed.

Next, distortions are corrected for the optical system of the digital camera for each image taking into account the distortion coefficient and image scaling factor, as shown in FIG. 7. If we compare the images in FIG. 7, it is seen that the objects shown in the image of FIG. 7 (a) have distortions of geometric shapes that are eliminated as a result of correction in the image of FIG. 7 (b).

In the next step, each image is cropped to the borders of its pentagonal section, as shown in FIG. 8, for the image received from the sensor 9. This operation is performed by FPGA.

Further, using the means of the ARM processor, in the areas of overlapping images coming from neighboring sensors, they search for singular points and construct their descriptions in the form of value vectors based on the SURF method. In FIG. 9, rectangles show the area of overlapping images coming from the sensors 5 (a) and 9 (b) of adjacent digital cameras, and an example of determining pairs of singular points and comparing them.

Using information on the relative position of pairs of singular points on adjacent images, images are deformed based on affine transformations, which allows the boundaries of images obtained from sensors of adjacent digital cameras to be combined, as shown in FIG. 10 for images received from adjacent sensors 5 and 9.

Then, using the dodecahedron model, address matrices are calculated that determine the location of an individual pixel of each image of the corresponding dodecahedron face in the Cartesian coordinate system.

Convert the Cartesian coordinates to spherical to obtain the projection of the image on the sphere inscribed in the dodecahedron (figure 5). After that, the equidistant cylindrical projection of the image of the area of the sphere corresponding to each of the 12 sensors is constructed. A fragment of the projection of a spherical image onto a plane for images obtained from adjacent sensors 5 and 9 is shown in FIG. eleven.

In the next step, each image in accordance with the addresses obtained during the calculation, which determine the location of a single pixel in each image, is stored in the memory of the central computing node, in which, using the software for constructing an equidistant cylindrical projection of the sphere, a composite panoramic image of the room shown in FIG. . 12.

To obtain a composite panoramic image for other types of regular polyhedra, for example, a dodecahedron, hexahedron, icosahedron, octahedron or tetrahedron, it is also necessary to use the transformation of Cartesian coordinates to spherical to obtain the projection of the image onto a sphere inscribed in the corresponding polyhedron, with the subsequent construction of an equidistant cylindrical image projection.

Thus, the possibility of implementing the method with obtaining the claimed technical result - high visual quality of a digital panoramic image of high resolution - is proved.

Claims (2)

1. A method of generating a digital panoramic image from several sources, including processing images obtained from cameras located on the edges of a regular polyhedron, by cropping each image below the section boundary in the form of a regular polygon, forming a composite spherical panoramic image with correction of the received images, characterized in that they first correct the images obtained from cameras located on the faces of a regular polyhedron based on the vector model hydrochloric field patterns by correcting distortions caused by the lens of each digital camera with the distortion coefficient and the coefficient zooming; cropping images received from the sensor of each camera is carried out by determining the ownership of each pixel of the input image to the correct n-gon inscribed in it, where n is from 3 to 5, then pairs of singular points are calculated in the areas of image overlap using the SURF method and additionally based on affine transformations carry out non-linear deformation of each cropped image according to pairs of singular points in the areas of overlapping images coming from adjacent digital cameras; then, to form a composite panoramic image, address matrices are calculated that determine the location and belonging of a single pixel of each image to the faces of the regular polyhedron in the Cartesian coordinate system, then the projection of the regular polyhedron onto the sphere inscribed in it is built, and then, using the software for constructing the equidistant cylindrical projection of the sphere, stitching images. 2. The method of forming a digital panoramic image according to claim 1, characterized in that the regular polyhedron can be made in the form of a dodecahedron, hexahedron, icosahedron, octahedron or tetrahedron.

Images