RU2797757C1 - Device and method for obtaining image sets - Google Patents

Abstract

FIELD: devices and methods for obtaining sets of images.

SUBSTANCE: self-aligning device for obtaining sets of images comprises N cameras, where N≥2, optically coupled through N-1 beam splitters, which would allow placing cameras with any mounts on it, namely one or more reference cameras that capture a good quality image, and one or multiple target cameras capturing low quality image. The cameras are mounted on adjustable holders. The alignment module controls the holders by aligning the system using characteristic points/elements captured in the scene being shot, on the alignment target or using the alignment target simulation module, in order to initially match the images as accurately as possible due to shifts, tilts and rotations of the cameras and thereby converge to a minimum of post-processing for the final combination of images. The cameras are controlled by the control module. Image sets collected with different settings are used to train image enhancement models.

EFFECT: preservation of alignment during operation, obtaining sets of images for different scenarios, with applicability to moving scenes and scenarios for removing blur.

26 cl, 6 dwg

Description

Technical field

The present invention relates to devices and methods for obtaining image sets consisting of one or more low quality images and its corresponding one or more high quality images obtained from two or more different digital cameras or video cameras (hereinafter referred to as cameras) used in problems of improving image quality using neural networks that are trained to "bring" the original image (low quality, "input") to the standard (high quality image, English - ground-truth, "correct answer"). A neural network trained on such sets of images can be used at the output of an image capture device to obtain high quality images even if low quality images are captured.

Description of the Prior Art

The main requirement for such devices and methods for obtaining sets of images is that the images in the set must match each other so well, i.e. have minimal or required geometric and photometric differences so that they can be used without post-processing, or with minimal post-processing for training neural network models that solve a number of image quality improvement tasks, such as noise reduction, resolution upscaling, detail upscaling, deblurring, demosaicing and creating an image processing operations pipeline (ISP).

The first problem in fulfilling these requirements is due to the difference in the optical characteristics of the cameras. Even when providing optical conjugation (combination of optical axes in space) of cameras using a beam splitter, the difference in optical characteristics will lead to a difference in the depth of field of the imaged space and optical aberrations. Therefore, alignment of images in a set without post-processing is possible only to a certain extent.

The second problem is the timing of the images. In order for the mutual positions of objects in the images to correspond to each other, it is necessary to ensure synchronized image capture (simultaneous or with a specified delay between cameras). Otherwise, the mutual positions of the objects may differ, for example, due to their movement.

Another challenge is to support different combinations of cameras with different optics and receivers, such as industrial cameras, smartphone cameras, digital SLR/mirrorless cameras.

The following approaches are used to solve the above problems.

1. Obtaining sets of images by artificially degrading the image quality.

Artificial image quality degradation (degradation) is often used to generate image sets: low quality images are obtained from high quality images by applying a degradation model. Neural networks trained on such synthetic data tend to remember the degradation model, their applicability to real data is very weak, especially in tasks where pixel-by-pixel improvement in image quality is required (down to the smallest details with high resolution). To implement this approach, an accurate and realistic image degradation model is required, while the actual degradation processes are usually unknown. To create image sets, high-quality source images are required: they must not contain features due to the optics with which they were taken. Also, high quality images can be obtained by rendering, but this is quite a difficult task.

2. Acquisition of sets of images using a single camera by shooting in quick succession with different quality settings.

Sets of images taken from the same camera by shooting in quick succession with different settings are only applicable for a limited number of tasks, such as detail enhancement or noise reduction, and only for almost static scenes. Training on such sets requires a complex neural network architecture that could compensate for the poor matching between images that inevitably occurs if the scene being filmed changes during shooting.

3. Obtaining sets of images using multiple cameras.

3.1 Obtaining sets of images using several adjacent cameras.

The main disadvantage of this approach is that the cameras see the same scene at different angles from different points in space (the optical axes of the cameras cross in space). This leads to a discrepancy between the mutual positions of objects located at different distances from the cameras in the images, which is especially critical in the case of scenes extended in depth. The fields of view of adjacent cameras overlap only partially. In this case, it is impossible to compensate for the difference between images by global transformations, and local transformations can lead to degradation of details. Models trained on image sets for which pixel-by-pixel alignment is not possible are not suitable for improving fine image details and are applicable to a limited number of scenarios.

3.2 Obtaining sets of images using two optically coupled cameras.

The use of two cameras, the optical axes of which are coupled in space with the help of a beam splitter, potentially makes it possible to achieve better alignment between images than options 2 and 3.1. In well-known works using this approach, due attention is not paid to a good initial alignment of images by adjusting the position of the cameras in space and their synchronization, and complex post-processing is used for the final alignment. However, as you know, the less post-processing contains data, the better they are for training models, since complex post-processing can introduce geometric distortions into the processed image and disrupt the natural noise profile. If the cameras are poorly synchronized, an attempt to combine images using post-processing leads to the appearance of artifacts on moving objects. Due to the impossibility of pixel-by-pixel image alignment due to poor initial camera alignment, models trained on such image sets are unsuitable for improving fine image details and are applicable to a limited number of scenarios. In addition, this approach requires a static scene (in the absence of synchronization) and a complex neural network architecture.

Thus, a common problem of the known approaches is the lack of diversity of the collected data. Basically, all the data is collected for one task - to increase the resolution of the image. The existing sets of images with well-aligned images are small and obtained for quasi-static scenes in laboratory conditions for some one set of cameras. It is difficult to obtain similar data outside the laboratory conditions and for various combinations of cameras, because due to mechanical vibrations and movement of lenses during refocusing, misalignment of the optical axes of the cameras can occur. Ordinary cameras (SLR cameras, smartphone cameras) are especially susceptible to vibrations, which do not have the possibility of reliable fixation, unlike industrial cameras, which have several mounting holes.

These problems could be solved by a universal device that provides synchronization and alignment of the cameras, as well as maintaining the alignment during operation. A self-aligning system would greatly simplify the replacement of a camera of one type with a camera of another type, would provide realignment of the system in case of mismatches during operation, and would allow collecting a variety of image sets with different scenes and under different conditions.

State of the art

The prior art document "ImagePairs: Realistic Super Resolution Dataset via Beam Splitter Camera Rig", Microsoft/arXiv:2004.08513v1, publication date 04/18/2020, which provides two cameras with different radiation receivers and different optics (low and high resolution camera), installed in adjustment mounts with 6 degrees of freedom, the optical axes of which are coupled through a beam splitter. The combination of images from cameras due to alignment is performed in several stages using a number of alignment targets. Distortion is corrected in both captured images, for a more accurate alignment, a homography transformation is applied to one of them, and additionally local matching is performed based on patches, i.e. parts of the image. Image pairs are used to train models for image upscaling, image upscaling, image processing pipeline (ISP) operations. The disadvantages of the known solution are the use of complex post-processing (including with different processing parameters for each local neighborhood) for the final alignment of images, as well as the lack of synchronization between cameras, which leads to a mismatch in the position of moving objects between images, which, in turn, leads to to the appearance of geometric artifacts caused by complex post-processing.

Also known from the prior art is the document "Capturing ground truth super-resolution data" from Karlsruhe Institute of Technology/Fraunhofer IOSB/DOI: 10.1109/ICIP.2016.7532872, publication date 09.2016, in which A laboratory model is described, consisting of two cameras with identical radiation receivers and different optics (wide-angle lens and telephoto zoom lens), one of which is mounted on a tilt-and-turn table and whose optical axes are coupled through a beam splitter. Combination of images from cameras due to alignment is performed by compensating for rotation in the plane of the radiation receiver using a tilt-rotary table and aligning the scales with a zoom lens. The remaining shifts are compensated by the Lucas-Kanade algorithm during post-processing. Pairs of images obtained using such a setup are used to train models for image upscaling tasks. The disadvantages of the known solution are that the optical axes of the cameras are not aligned in space, which means that the solution is applicable only for quasi-flat scenes, as well as a rough initial alignment of images due to camera alignment, which requires complex post-processing based on estimating the light flux for the final alignment of images. .

The prior art document "DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks" ("DSLR-quality photos on mobile devices with deep convolutional networks") from ETH Zurich/arXiv:1704.02470v2, publication date 09/05/2017, which describes a method for obtaining pairs of images using two cameras with different radiation receivers and different optics (SLR cameras and smartphones), which are located next to each other and have partially overlapping fields vision. The captured images are combined using a homography transformation calculated on the basis of matching the characteristic points of the scene found using the SIFT (scale invariant feature transformation) algorithm. Image pairs are used to train models for image enhancement tasks. The disadvantages of the known solution is that the images cannot be combined in depth, since the cameras have different viewing angles and spatial positions.

Document WO 2020/246861 A1, publication date 10/12/2020, is known from the prior art, disclosing "Method and apparatus for training neural network model for enhancing image detail" ("Method and apparatus for training a model based on neural networks to increase image detail") , according to which pairs of images are obtained by one camera by quickly sequentially capturing a pair of images with different quality settings. Image pairs are used to train models for image detail enhancement and noise reduction tasks. The disadvantage of the known solution is that it is suitable for almost static scenes and a moderate difference in the exposures of the captured images, as well as the fact that training on such pairs of images requires a complex neural network architecture that would allow compensating for poor alignment of sequentially captured frames.

Positive results of the invention

In accordance with the present invention, support is provided for various types of cameras in obtaining sets of images and their alignment, as well as maintaining the alignment during operation.

In accordance with the present invention, it is also possible to obtain sets of images for different scenarios, such as upscaling, upscaling, demosaicing.

In accordance with the present invention, the applicability of the invention to moving scenes and deblur scenarios is also provided.

In accordance with the present invention, there is also no post-processing for the final alignment of images, or the application of only global transformations to the reference images in the set (which avoids the occurrence of local geometric artifacts), while the images fed to the input of the neural network remain in their original form.

It also provides the ability to align the system regardless of the scene, making it possible to apply the invention to a wide range of problems.

The essence of the invention

A self-aligning device is proposed for obtaining sets of images using N digital cameras, where N≥2, optically coupled through N-1 beam splitters, which would allow placing on it any N cameras with any mounts, namely, one or more reference cameras, that capture good quality images and one or more target cameras that capture poor quality images. The cameras are mounted on adjustable holders that provide linear and rotary movements. The alignment module controls the holders with high precision, adjusting the system using characteristic points captured in the scene being shot, on the adjustment target or using the adjustment target simulation module, in order to initially match the images as accurately as possible by means of the device due to shifts, tilts and rotations of the cameras and thereby minimizing post-processing for the final alignment of images. The cameras are controlled by the control module.

A method is also proposed for obtaining sets of images using this device, which can be used to train image quality enhancement models based on neural networks.

Embodiments

In one embodiment of the invention, an image set acquisition device is characterized, comprising: N digital cameras, where N≥2, including one or more reference cameras capturing good quality reference images and one or more target cameras capturing poor quality target images. mounted on said device and optically coupled with N-1 beamsplitters, with at least N-1 cameras mounted on automatically adjustable holders, an alignment module used to align at least N-1 cameras out of N cameras based on N images scenes, wherein the adjustment is carried out by actuating adjustable holders on which the mentioned N-1 cameras are installed, a digital camera synchronization system for taking images, a control module for controlling digital cameras and providing communication between the individual parts of the device and obtaining sets of N images, wherein the control module instructs the alignment module to align at least N-1 cameras based on the scene images.

In a further embodiment of the invention, the control module further performs color calibration of the N-1 cameras to match the colors of the reference images to those of the target images.

In an additional embodiment of the invention, the control module additionally calculates the coefficients of the necessary polynomial transformations to bring the reference images into correspondence with the target images in spatial coordinates, and applies polynomial transformations to the said reference images for their more accurate spatial alignment with the target images.

In an additional embodiment of the invention, optically coupled cameras with the same lenses and the same radiation receivers form identical optical images of the same scene with good spatial alignment in the central region of the image.

In an additional embodiment of the invention, optically conjugated cameras with the same lenses and different radiation receivers form optical images of the same scene with different parameters with good spatial alignment when all images are scaled to one in the central image area.

In an additional embodiment of the invention, optically conjugated cameras with different lenses and the same radiation receivers form optical images of the same scene with different parameters with good spatial alignment when all images are scaled to one in the central image area.

In an additional embodiment of the invention, optically coupled cameras with different lenses and different radiation detectors form optical images of the same scene with different parameters with good spatial alignment when all images are scaled to one in the central image area.

In a further embodiment of the invention, at least one optical filter may be inserted in front of at least one camera to independently change the light output by attenuating either spectral or polarization filtering in at least one optical channel of the device.

In another embodiment of the invention, an image set acquisition device is characterized, comprising: N digital cameras, where N≥2, including one or more reference cameras capturing good quality reference images and one or more target cameras capturing poor quality target images. mounted on said device and optically coupled by means of N-1 beam splitters, with at least N-1 camera mounted on an automatically adjustable holder, a target simulation module that forms virtual images of the alignment target at two predetermined distances and contains two focusable alignment target simulators , which are optically coupled through a beam splitter, each of which consists of a light source, a test object and a collimating system, an alignment module used to align at least N-1 cameras out of N cameras based on N images of an imaginary alignment target, and the alignment is performed by actuating adjustable holders on which said N-1 cameras are mounted, a camera synchronization system for capturing images, a control module for controlling cameras and providing communication between individual parts of the device, configured to issue synchronization commands to the camera synchronization system when capturing a set images, and the control module is configured to issue commands to the adjustment module to align at least N-1 cameras from among N cameras based on images of the imaginary adjustment target, set shooting parameters for the cameras.

In a further embodiment of the invention, the control module performs color calibration of the N-1 cameras to match the colors of the reference images to the colors of the target images.

In an additional embodiment of the invention, the control module additionally calculates the coefficients of the necessary polynomial transformations to bring the reference images into correspondence with the target images in spatial coordinates, and applies polynomial transformations to the said reference images for their more accurate spatial alignment with the target images.

In an additional embodiment of the invention, optically coupled cameras with the same lenses and the same radiation receivers form identical optical images of the same scene with good spatial alignment in the central region of the image.

In an additional embodiment of the invention, optically conjugated cameras with the same lenses and different radiation receivers form optical images of the same scene with different parameters with high spatial alignment when all images are scaled to one in the central image area.

In an additional embodiment of the invention, optically coupled cameras with different lenses and the same radiation receivers form optical images of the same scene with different parameters with high spatial alignment when all images are scaled to one in the central image area.

In an additional embodiment of the invention, optically coupled cameras with different lenses and different radiation receivers form optical images of the same scene with different parameters with high spatial alignment when all images are scaled to one in the central image area.

In a further embodiment of the invention, at least one optical filter may be inserted in front of at least one camera to independently change the light flux in at least one optical channel of the device.

In another embodiment of the invention, a method for obtaining sets of images is characterized, comprising the steps of focusing to the required distance N digital cameras, where N≥2, forming a system of digital cameras and requiring focusing, optically coupled using a beam splitter; aligning a system of N digital cameras, where N≥2, optically coupled with N-1 beam splitters, said system of N digital cameras including one or more reference cameras that capture good quality reference images, and one or more target cameras that capture low quality target images using the selected algorithm to achieve good image registration in the central region where optical aberrations are small; using optically coupled digital cameras form optical images of the same scene with good spatial alignment in the central region of the image; receiving a set of N images from a system of N digital cameras synchronized by the start time of shooting with different shooting parameters.

In an additional embodiment of the invention, the synchronization parameters are set in such a way that at least one camera out of N cameras has a long exposure, and the remaining cameras out of N cameras have a short exposure, the beginning of which is shifted relative to the start of the long exposure by the required amount.

In an additional embodiment of the invention, a set of images of a calibration color target is preliminarily registered; on the basis of these images, a color correction matrix is calculated, which makes it possible to bring the colors of all images in the set to the colors of one image in the set; based on the calculated color correction matrix, all registered images in the set are color corrected except for one.

In a further embodiment of the invention, the coefficients of the necessary polynomial transformations are calculated to bring the reference images in the set to match the target images in spatial coordinates; applying said polynomial transformations to the reference images in the set to more accurately spatially match them to the target images.

In an additional embodiment of the invention, using optically coupled cameras with the same lenses and the same radiation receivers, optical images of the same scene with identical parameters are formed with good spatial alignment in the central region of the image.

In an additional embodiment of the invention, using optically coupled cameras with the same lenses and different radiation receivers, optical images of the same scene with different parameters are formed with good spatial alignment when all images are scaled one by one in the central image area.

In an additional embodiment of the invention, using optically coupled cameras with different lenses and the same radiation detectors, optical images of the same scene with different parameters are formed with good spatial alignment when scaling all images one by one in the central image area.

In an additional embodiment of the invention, using optically coupled cameras with different lenses and different radiation receivers, optical images of the same scene with different parameters are formed with good spatial alignment when scaling all images one by one in the central image area.

In a further embodiment of the invention, the luminous flux for at least one of the digital cameras is changed by means of an optical filter installed in front of at least one of the digital cameras, thereby attenuating or spectral filtering or filtering by polarization of the light flux.

In yet another embodiment of the invention, an image set acquisition device is characterized, comprising:

two digital cameras, including one reference camera capturing good quality reference images and one target camera capturing poor quality target images, fixed on said device and optically coupled with a beam splitter, wherein

at least one camera is mounted on automatically adjustable holders,

an adjustment module used to align at least one camera based on two images of a scene, the adjustment being carried out by actuating adjustable holders on which said cameras are mounted,

digital camera synchronization system for taking pictures,

a control module for controlling digital cameras and providing communication between individual parts of the device and obtaining sets of images,

wherein the control module instructs the alignment module to align at least one camera based on the images of the scene.

Brief description of the drawings

The above and other features and advantages of the present invention are explained in the following description, illustrated by drawings, in which the following is presented:

Fig. 1 schematically illustrates the proposed device for obtaining sets of images in a dual camera configuration.

Fig. 2 illustrates a functional diagram of the claimed device in a two-camera configuration that provides automatic alignment to the observed scene.

Fig. 3 illustrates the operation of the adjustment module.

Fig. 4 illustrates an example of an alignment target with two planes.

Fig. 5 illustrates the optical scheme of the claimed device in a configuration with two cameras, realizing the alignment mode on an imaginary target.

Fig. 6 schematically illustrates the proposed device in an N-chamber configuration.

Detailed description of the invention

Fig. 1 schematically illustrates the proposed device for obtaining sets of images, containing two digital cameras (2 and 3) optically coupled through a beam splitter (1). Depending on the problem being solved, various combinations of cameras are possible, namely: cameras with the same lenses and the same matrix radiation detectors, cameras with the same lenses and different radiation detectors, cameras with different lenses and the same radiation detectors, cameras with different lenses and different radiation detectors. In this case, at least one camera is mounted on adjustable holders (21), controlled by the adjustment module (4), and providing linear and rotary movements, driven by the adjustment module (4). Variants are also possible when both chambers (2 and 3), or one chamber and a beam splitter, or both chambers (2 and 3) and a beam splitter (1) are mounted on adjustable holders (21). The adjustment module (4) performs automatic alignment of images from two cameras by adjusting according to an algorithm based on a comparison of the position of characteristic points on two images. These points can be taken from the scene (less accurately), or from a two-plane alignment target (more accurately, but a physical target is needed), or from an imaginary target formed by the target simulation module (9). The cameras are synchronized for shooting using a synchronization system (6) (for example, using an electrical and / or optical trigger, network time protocol), which allows you to coordinate the start of shooting between cameras for shooting in the master camera-slave camera mode in accordance with the settings, received from the control module directly, or through the leading camera, depending on the implementation of the synchronization system (6). In the case of using a synchronization system (6), a signal for shooting from the control module (5) is sent to one of the cameras (the so-called master camera). This camera issues a command to shoot a second camera (the so-called slave camera, one or more) either through a trigger or via a network protocol. Figure 1 shows camera 1 (2) as the master camera, which issues a shooting command to the slave camera 2 (3) by applying a synchronization signal to the synchronization system (6), which then issues a trigger signal to camera 2 (3).

It is also possible to shoot without using the synchronization system, in which case both cameras will receive a shooting command directly from the control module (5). But in this case, the images will not be consistent in time, which is especially noticeable on moving objects. This option is only suitable for static scenes. The synchronization system allows you to take into account the difference in the processing time of the control signal / processing of the cameras and correctly place the shooting delays on each camera so that in the end they shoot at the same global time point or with the required delay relative to each other.

Optical filters (7 and 8) can be inserted into the optical channels of the device to independently change the light flux coming to the cameras. The control module (5) controls the cameras (2 and 3), controls the synchronization system (6), provides communication between the individual parts of the device, transfers the captured images from both cameras (2 and 3) to the adjustment module (4), which controls the adjustable holders ( 21) cameras.

The operation of this device is a method for obtaining image sets with various combinations of lenses and/or radiation receivers and/or filters with adjustable timing and, if necessary, very simple post-processing of only the reference images in the image set, which may include the application of color correction and polynomial transformations . The way to get sets of images is that:

First, cameras that require focusing are focused to the required distance. The device is aligned using the selected algorithm to achieve good image alignment in the central region where optical aberrations are small. Adjustment is repeated as necessary during the shooting process.

Using the same optically coupled cameras (2 and 3), at least two optical images identical in terms of parameters (with the same magnification, resolution, etc.) of the same scene with good spatial alignment are formed.

Recording of sets of images is carried out with synchronization in time of the start of shooting with different shooting parameters (exposure, gain of the analog-to-digital converter). Synchronization parameters are set based on the task: at least two cameras (2 and 3) shoot synchronously, or at least one of the cameras has a long exposure, and the rest have a short one, the beginning of which is shifted relative to the beginning of the long exposure by the required amount. The resulting image sets are used for tasks such as image enhancement, deblurring, and noise reduction.

In an additional embodiment of this method, an additional color rendering calibration is performed, while a set of images of a calibration color target is recorded, then a color correction matrix is calculated based on these images, which allows to bring the colors of all images in the sets to the colors of one image in the set.

In an additional embodiment of this method for a set of images, the coefficients of the necessary polynomial transformations are calculated to bring the images of the reference cameras in the set into correspondence with the images of the target cameras in the set in spatial coordinates, polynomial transformations are applied to the images from the reference cameras in the set for more accurate image alignment.

In an additional embodiment of this method, using optically coupled cameras (2 and 3) with the same lenses and different radiation detectors, optical images of the same scene with different parameters are formed with different resolutions, with good spatial alignment when scaling all images under one in center area of the image. The resulting image sets are used for tasks such as image resolution enhancement.

In an additional embodiment of this method, using optically coupled cameras (2 and 3) with different lenses and the same radiation detectors, optical images of the same scene with different magnifications are formed, different in parameters, with good spatial alignment when scaling all images under one in center area of the image. The resulting image sets are used for tasks such as image resolution enhancement.

In an additional embodiment of this method, using optically coupled cameras (2 and 3) with different lenses and different radiation detectors, optical images of the same scene with different parameters, having different magnifications and different resolutions, are formed with good spatial alignment when scaling all images under one in the central area of the image. The resulting image sets are used for tasks such as image upscaling.

In an additional embodiment of this method, the light flux for at least one of the cameras (2 and/or 3) is changed using a filter/filters (7 and/or 8) installed in the optical channels of the device in front of the camera/cameras (2 and/or 3), thereby performing attenuation or spectral or polarization filtering.

Thus, as shown in FIG. 1, the first key feature of the present invention is the presence of an alignment module that automatically aligns the device to the observed scene.

In FIG. 2 shows a functional diagram of the claimed device in a configuration with two cameras (2 and 3), which provides automatic adjustment to the observed scene using an adjustment module. The scheme shown in Fig. 2 is different from the circuit shown in FIG. 1, the absence of a target simulation module (9). In FIG. 2 numbers indicate the following:

2A - Scene with objects, or with an adjustment target, including two planes.

2B - The light beam carrying the image enters the beam splitter (1), which divides it into two beams.

2B - The beams are sent to the first chamber (2) and the second chamber (3), respectively, where they form images of the scene on the matrix radiation receivers of each of the chambers (2 and 3).

2D - On command from the control module (5), two images are registered and transferred from each of the cameras (2 and 3) to the control module (5).

2D - The control module (5) transmits the registered images to the alignment module (4) if the user intends to align the device; the adjustment module (4) detects the characteristic points of the observed scene present in both registered images.

In this case, the control module (5) allows the user to select the adjustment mode:

(1) the mode of adjustment by characteristic points (elements) found on the scene uses the characteristic points belonging to objects located in the field of view of the device, automatically searching for common characteristic points present in both images; the advantage of this mode is that it does not require special adjustment targets to be placed in front of the device, and characteristic points are taken from objects from the field of view of cameras (2 and 3); however, for some scenes it can be difficult to find well-detected characteristic points, which narrows the scope of this mode; it is also possible that these points will be unevenly distributed in the scene space, which means that the accuracy of calculating the displacement and rotation of cameras (2 and 3) based on such sparse uneven points will be lower;

(2) adjustment mode using an adjustment target containing a previously known pattern, on which characteristic elements are easily determined, and consisting of two planes at different distances, placed in front of the device; this mode requires placement of an adjustment target with a previously known pattern in front of the device, on which the search for characteristic points/elements will be performed; The advantage of this mode is that the adjustment target is designed in such a way that it is very easy to detect it in the field of view of the device and it provides uniform filling of the field of view with characteristic points that can be determined with high accuracy, since the pattern (i.e., the pattern on the target ) is known in advance and selected specifically for this task, which means that the accuracy of estimating displacements using such a target will be higher; The disadvantage of this mode is that for it it is necessary to have a physical adjustment target, which must be specially placed in the scene space;

(3) the virtual target adjustment mode uses the adjustment target simulation module (9) (the mode is available only for the configuration of Fig. 1) to build a virtual target image at an arbitrary place in the scene; The advantage of this mode is that the shape of the imaginary adjustment target is chosen in such a way that it is very easy to detect it, and using this approach, it is possible to align the system regardless of the observed scene (even when there are no objects in the field of view at all), which increases the reliability of operation. way.

The adjustment module (4) calculates, according to the mode selected by the user, the difference in the coordinates of the characteristic points between the two images; if this difference exceeds a predetermined threshold, the adjustment module (4) calculates the necessary shifts and rotations of the cameras to minimize it.

The feature point coordinate comparison mechanism is described in detail below. In general, the comparison is carried out by calculating the differences in the coordinates of these points, as well as the ratio of the distances between these points in two images. The direction of movement of the cameras is chosen in such a way as to minimize the difference in coordinates.

2E - The alignment module (4) transmits the necessary control commands to the movable holders (21) of the cameras (or cameras (2 and 3) and the beam splitter (1)) in order to align the optical axes, the rotation angles of the radiation receivers and the entrance pupils of the cameras (2 and 3 ).

After the alignment is completed, the user starts the procedure for obtaining a set of images with the settings necessary for the problem being solved; further, the resulting set of images is used to train models based on neural networks to improve the quality of images.

Thanks to the above actions, it is possible to automatically adjust for various combinations of cameras (2 and 3) and maintain the alignment (ie, realign as necessary) during operation.

Returning to FIG. 1, we note that the second key feature of the present invention is the presence of a module (9) for simulating an adjustment target, which makes it possible to align the system regardless of the scene.

As indicated above, in FIG. 1 shows a functional diagram of the claimed device in a configuration with two cameras (2 and 3), with a detailed alignment target simulation module (9) including two optically coupled target simulators (11 and 12) building two virtual images of the target at different distances. in front of the device, which is labeled as follows:

1A - If the user selects the adjustment mode according to the imaginary target, the input window of the device is closed by the first opaque screen (13), and the path to the target simulation module (9) is opened by removing the second opaque screen (14) from the beam path.

1B - The alignment module (4) sends commands to the target simulation module (9) to turn on the light sources of the target simulators (11 and 12) and to focus to create imaginary targets at the required distances.

1B - The light beam that transfers images of two imaginary targets from the target simulation module (9) enters the beam splitter (1), where it is divided into two beams.

1D - Beams from beam splitters (1 and 10) arrive at chamber 1 (2) and chamber 2 (3), respectively.

1D - On command from the control module (5), two images are registered and transferred from each of the cameras (2 and 3) to the control module (5).

1E - The control module (5) sends images to the alignment module (4) if the user intends to align the system. The adjustment module (4) calculates the required direction of movement of the cameras (2 and 3) based on the characteristic points detected according to the adjustment mode selected by the user.

1G - The adjustment module (4) transmits the necessary control commands to the movable adjustable holders (21) of the cameras (or cameras (2 and 3) and the beam splitter (1)) in order to align the optical axes, the rotation angles of the radiation receivers and the entrance pupils of the cameras (2 and 3).

After the alignment is completed, the light sources of the module (9) of the simulated target are turned off; the screen (14), blocking the target simulation module, is introduced into the path of the rays; the screen (13) blocking the scene is removed from the path of the rays. The user starts obtaining sets of images with the settings necessary for the task being solved; further, the resulting set of images is used for deep learning of models to improve the quality of images.

Thanks to the above actions using the target simulation module, it is possible to align the device regardless of the scene.

In FIG. 3 shows the operation of the adjustment module (3) in the configuration of the device with two chambers (2 and 3).

Automated adjustable holders must provide at least 6 degrees of freedom for image alignment due to camera alignment (2 and 3):

1) Camera 1 (2) offset along the Y1 axis or camera 2 (3) offset along the Y2 axis;

2) Camera shift 1 (2) along the X1 axis or camera shift 2 (3) along the X2 axis;

3) Offset camera 1 (2) along the Z1 axis or offset camera 2 (3) along the Z2 axis;

4) Rotation of camera 1 (2) around the X1 axis or rotation of camera 2 (3) around the X2 axis (or rotation of the beam splitter (1) around the Z0 axis);

5) Rotate camera 1 (2) around the Y1 axis or rotate camera 2 (3) around the Y2 axis (or rotate the beam splitter (1) around the Y0 axis)

6) Rotate camera 1 (2) around the Z1 axis or rotate camera 2 (3) around the Z2 axis

It should be noted that rotations of the beam splitter (1) are possible by issuing commands to rotate the beam splitter (1), but are not desirable, since it is preferable to have a common fixed component for the optical channels of the device.

The control module (5) initiates the shooting of the scene by the cameras and sends the received images to the adjustment module (4). The adjustment module (4) calculates the mismatch between the XYZ coordinate systems of cameras (2 and 3) and controls the adjustable holders to align them by issuing commands to shift/rotate camera 1 (2) and commands to shift/rotate camera 2 (3).

Returning to the three adjustment modes mentioned above, the following should be noted.

(1) The first option for selecting the adjustment mode by the user is the adjustment mode based on characteristic points found on the scene.

Scene-based alignment performed by the alignment module (4) uses feature points found in the scene. This requires a scene extended in depth with characteristic points evenly distributed within the field of view of the device.

The control module (5) focuses the cameras at the required distance. If the cameras (2 and/or 3) do not have autofocus or electronic focus control, manual focusing is done by the user.

The adjustment module (4) provides the following adjustment steps:

1) Detection of characteristic points corresponding to each other present in both images;

2) Calculation of the scaling factor as the ratio of the distances between the corresponding points on the first and second images. This coefficient can be taken into account when calculating offsets, or one of the images can be resized to adjust the scale.

3) Setting the same difference in vertical coordinates y of coinciding characteristic points Δy using a vertical shift of one of the cameras (2 or 3), followed by minimization of Δy by rotating one of the cameras (2 or 3) relative to the horizontal axis (or a similar rotation of the beam splitter (1)) .

4) Setting the same difference in the horizontal coordinates x of the coinciding characteristic points Δx using a horizontal displacement of one of the cameras (2 or 3), followed by minimization of Δx by rotating one of the cameras (2 or 3) around the vertical axis (or a similar rotation of the beam splitter (1)) .

5) Rotation of one of the cameras around the optical axis to match the angles of rotation of the radiation receivers of the cameras.

6) Setting the optimal axial position by moving one of the cameras (2 or 3) along its optical axis in such a way as to achieve a constant zoom factor within the depth of the scene.

In this case, steps 2)-6) are not necessarily performed in the specified order and can be repeated in iterations.

Thanks to the above actions, it is possible to automatically align images on the scene without using targets.

(2) The second option for selecting the adjustment mode by the user is the adjustment mode on a real target. Alignment to a real target provides higher accuracy because the alignment targets are guaranteed to be evenly distributed within the field of view of the device and are known in advance.

This requires a composite alignment target, including two targets mounted at different distances from the device, as shown at the top of FIG. 4.

An anterior plane target closer to the device containing an alignment pattern (eg circles) allows a rear plane target further away from the device to be seen with an alignment pattern. The adjustment pattern is chosen in such a way that it is easy to detect and it is possible to calculate the coordinates of characteristic elements with high accuracy from it.

Adjustment targets are installed in such a way that the focusing plane of the cameras is between them, or on one of them.

The control module focuses the cameras at the required distance. If the cameras (2 and 3) do not have autofocus or electronic focus control, manual focusing is done by the user.

The adjustment module (4) in this mode provides the following adjustment steps:

1) Detection of characteristic elements of alignment patterns on images from both cameras (2 and 3) with high accuracy; in this case, each image contains adjustment patterns belonging to the adjustment targets of the front and rear planes, for example, the front plane target occupies II and IV coordinate quarters, and the rear plane target occupies I and III coordinate quarters in the field of view of the cameras, as shown in Fig. 5; this is required to align the optical axes of the cameras (2 and 3) in space.

On FIG. 4 in the lower part shows a possible view of images from non-adjusted cameras, which shows that the adjustment targets in the fields of view of the cameras (2 and 3) are presented in an unequal way - they are shifted and / or tilted relative to each other.

2) Calculation of the scaling factor as the ratio of the distances between the corresponding elements of the adjustment pattern on the first and second images for one of the targets (either the front plane target or the rear plane target). This coefficient can be taken into account when calculating the offsets, or one of the images can be resized to fit the image scales.

3) Setting the same difference in vertical coordinates Δy for the corresponding elements of the adjustment pattern by vertical displacement of one of the cameras (2 or 3), followed by minimization of Δy by rotating one of the cameras (2 or 3) relative to the horizontal axis (or rotating the beam splitter (1)).

4) Setting the same difference in horizontal coordinates Δx for the corresponding elements of the adjustment pattern by horizontal displacement of one of the cameras (2 or 3), followed by minimization of Δx by rotating one of the cameras (2 or 3) relative to the vertical axis (or rotating the beam splitter (1)).

5) Rotation of one of the cameras (2 or 3) around the optical axis to match the angles of rotation of the radiation receivers of the cameras (2 or 3).

6) Setting the optimal axial position by moving one of the cameras (2 or 3) along its optical axis in such a way as to minimize the difference in scaling factors for the front plane target and the rear plane target.

In this case, steps 2)-3) are not necessarily performed in the specified order and can be repeated in iterations.

Thanks to the above actions, greater reliability and accuracy are achieved compared to the first mode.

(3) The third option for selecting the adjustment mode by the user is the adjustment mode on an imaginary target.

On FIG. 5 illustrates the optical scheme of the claimed device, which implements the alignment mode on an imaginary target. This device is generally similar to the device according to the first and second options for selecting the adjustment mode by the user, but additionally includes an adjustment target simulation module (9), which simulates targets in the field of view of the camera system of the claimed device, moreover:

The target simulation module (9) contains two focusable alignment target simulators (2 or 3) optically coupled through beam splitters (1 and 10), each of which includes a light source, a test object and a collimating system. Target simulators (11 and 12) build virtual images of their test objects (targets) illuminated by light sources at two specified distances in front of the system of two cameras (2 and 3), thus simulating a target from two planes (front plane target and target rear plane). The distances at which virtual images of targets are built are chosen similarly to the previous case - so that the focusing plane of the cameras (2 and 3) is between them, or on one of them.

The shape of imaginary targets (test objects) is chosen so that they are easy to detect and can be used to calculate all the coordinate differences/ratios necessary for camera alignment. In particular, in FIG. 5 on its right side shows a possible display of ghost targets, shown as crosses for illustrative purposes, when the system is not aligned.

The adjustment module (5) controls the adjustable camera holders (21) using an algorithm that minimizes the difference in the position of the calibration targets on the images from the two cameras (2 and 3). The control steps are similar to those described above in relation to the second option for selecting the adjustment mode by the user.

At the time of adjustment, the opaque screen 2 (14) is removed from the path of the rays, the light sources of the simulators (11 and 12) are turned on. After the alignment, the light sources of the simulators are turned off, and the module (9) of the simulation of the adjustment target is covered by a movable opaque screen 2 (14).

At the same time, there is an additional embodiment of the third option for selecting the adjustment mode by the user, in which, for the period of adjustment, the opaque screen 2 (14) is removed from the path of the rays, the scene is covered with an opaque screen 1 (13), the light sources of the simulators (11 and 12) are turned on. After the alignment, the light sources of the simulators (11 and 12) are turned off, the module (9) of the adjustment target simulation is covered by a movable opaque screen 2 (14), and the opaque screen 1 (13) is removed from the beam path. Covering the scene for the duration of the alignment using screen 1 (13) facilitates the detection of imaginary alignment targets, which in this case will be observed against a dark background, and not on top of the real scene, as shown on the right side of FIG. 6 on the right and left, respectively.

Thanks to the above actions, the possibility of automatic camera alignment is achieved, regardless of the presence of characteristic points in the scene and does not require the use of physical targets.

Fig. 6 schematically illustrates the proposed device for obtaining sets of images containing N digital cameras, N>2. It should be noted that according to the principle of operation, the circuit shown in Fig. 6 is generally similar to the circuit shown in Fig. 1, however, is its modification, in which not two digital cameras (2 and 3) are involved, but N digital cameras (2, 3, 14, 15), indicated in FIG. 6 digits 1, 2, 3, ... N, with N>2, optically coupled through N-1 beam splitter (20). If necessary, to equalize the lengths of optical paths in the channels of the device, i.e. to compensate for the difference in the thickness of the material of the beam splitters (1, 19, 20) on the way from the entrance to the device to the chamber (2, 3, 14, 15), path difference compensators (17 and 18), which are plane-parallel plates, can be introduced into the channels of the device.

The number of cameras N that can be installed in the system is limited by technological considerations.

So, in particular, with an increase in the length of the optical path, aberrations accumulate in the material of beam splitters (1, 19, 20). For beam splitters in the form of plane-parallel plates, they are small and will be invisible for a reasonable length of the system. Aberration compensators can be used to compensate for aberrations.

Another limitation is the drop in system light transmission with the addition of each new beam splitter for pairs of cameras following it. This circumstance can be used to simulate the operation of cameras in low light conditions. However, if this is not required, the light transmission can be equalized for all cameras using attenuating filters.

In addition, to maintain the field of view of the camera system when adding new cameras and beam splitters, it is necessary that each subsequent beam splitter be several times larger in size than the previous one. In practice, this is difficult to achieve and beam splitters have a limited size, so the field of view of the system will decrease with increasing its length.

Possible additional embodiments of the invention

Optical images with the same or different shooting parameters with different magnifications, resolutions, etc. are formed depending on the specific task with different combinations of camera parameters in a set of cameras fixed on the device, namely: cameras with the same lenses and the same matrix radiation receivers, cameras with the same lenses and different receivers, cameras with different lenses and the same receivers, cameras with different lenses and different receivers.

An optical filter or filters (neutral density, spectral, polarization) can optionally be inserted in front of at least one camera to independently change the light flux in the optical channels of the claimed device. This may be necessary, for example, to simulate the operation of one of the cameras in low light - in this case, an attenuating light filter is placed in front of it.

Before registration of image sets, registration of a set of images of a calibration color target can additionally be performed, i.e. photographing a target containing samples of different colors. Based on these images, a color and/or brightness correction matrix is calculated, which allows you to bring the colors and brightness of all images in the set to the colors and brightness of one image in the set, and thereby compensate for the difference in color rendering and brightness of different cameras in the set of cameras attached to the device .

The coefficients of the necessary polynomial transformations can also be additionally calculated. Calculated transformation coefficients can be applied to images from reference cameras in the set for more accurate spatial matching of image sets, and due to the fact that polynomial transformations are global and not local, they are applied to the entire image, and not to its individual parts, which prevents the appearance of unwanted image artifacts, as described above in the description of problems not solved in the prior art.

Thanks to the present invention, it becomes possible to support various types of cameras and automatically align them to obtain image sets, and, most importantly, it becomes possible to maintain the alignment state during operation when registering an image set.

Post-processing of images using the present invention is insignificant, and is not applied to all images in the set, in particular, it is applied only to images from reference cameras, or not at all, depending on the difference between the characteristics of the cameras.

The present invention is suitable for various deep learning scenes and scenarios, in particular for moving objects and deblur scenario.

Thanks to the present invention, it becomes possible to combine images by aligning the cameras independently of the scene.

The present invention provides high (pixel) image registration accuracy for certain scenarios (depending on the degree of difference between the cameras).

With regard to the sets of images obtained in the implementation of the present invention, it should be noted that they have a high accuracy of image alignment (up to 1 pixel for individual scenarios); allow training models with simple, easily interpretable loss functions due to the fact that a low-quality image is fed to the input of the neural network, and the loss function during network training is a measure of the difference between the image at the output of the network and a higher quality reference image obtained from the reference camera; allow the use of models with a simple architecture, which are not required to compensate for a large difference between images; can be used for a wide range of tasks (resolution enhancement, noise reduction, image blur removal, etc.); carry little post-processing or can be used without post-processing (in some scenarios).

The present invention supports various cameras without a special mount, as the alignment module re-aligns during operation if necessary.

The present invention is suitable for obtaining image sets for various deep learning tasks, since it allows independent change in the light output in the channels of the device and synchronization between cameras.

Although the invention has been described in connection with some illustrative embodiments, it should be understood that the invention is not limited to these specific embodiments. On the contrary, the summary is intended to include all alternatives, corrections, and equivalents that may be included within the spirit and scope of the claims.

In addition, the invention retains all equivalents of the claimed invention, even if the claims change during the course of the application.

Claims (71)

1. Device for obtaining sets of images, containing: N digital cameras, where N≥2, including one or more reference cameras capturing good quality reference images and one or more target cameras capturing poor quality target images, fixed on said device and optically coupled with N-1 beam splitters , and at least N-1 camera is mounted on automatically adjustable holders, an adjustment module used to align at least N-1 cameras out of N cameras based on N scene images, the adjustment being carried out by actuating adjustable holders on which said N-1 cameras are mounted, digital camera synchronization system for taking pictures, a control module for controlling digital cameras and providing communication between the individual parts of the device and obtaining sets of N images, wherein the control module instructs the alignment module to align at least N-1 cameras based on the scene images. 2. The device according to claim 1, in which the control module additionally: performs color calibration of N-1 cameras to match the colors of the reference images to the colors of the target images. 3. The device according to any one of paragraphs. 1, 2, in which the control module additionally: calculates the coefficients of the necessary polynomial transformations to bring the reference images into correspondence with the target images in spatial coordinates, and applies the polynomial transformations to the said reference images for their more accurate spatial alignment with the target images. 4. The device according to any one of paragraphs. 1-3, in which additionally: Optically coupled cameras with the same lenses and the same radiation detectors form identical optical images of the same scene with good spatial alignment in the central region of the image. 5. The device according to any one of paragraphs. 1-3, in which additionally: Optically coupled cameras with the same lenses and different radiation detectors form optical images of the same scene with different parameters with good spatial alignment when all images are scaled to one in the central image area. 6. The device according to any one of paragraphs. 1-3, in which additionally: Optically coupled cameras with different lenses and the same radiation receivers form optical images of the same scene with different parameters with good spatial alignment when all images are scaled one by one in the central image area. 7. The device according to any one of paragraphs. 1-3, in which additionally: Optically coupled cameras with different lenses and different radiation detectors form optical images of the same scene with different parameters with good spatial alignment when all images are scaled to one in the central image area. 8. The device according to any one of paragraphs. 1-7, in which at least one optical filter can be inserted in front of at least one camera to independently change the light output due to attenuation or spectral filtering or polarization filtering in at least one optical channel of the device. 9. Device for obtaining sets of images, containing: N digital cameras, where N≥2, including one or more reference cameras capturing good quality reference images and one or more target cameras capturing poor quality target images, fixed on said device and optically coupled with N-1 beam splitters , and at least N-1 camera is mounted on an automatically adjustable holder, a target simulation module that forms virtual images of an alignment target at two specified distances and contains two focusable alignment target simulators that are optically coupled through a beam splitter, each of which consists of a light source, a test object and a collimating system, an adjustment module used to align at least N-1 cameras out of N cameras based on N images of a virtual alignment target, the adjustment being carried out by actuating adjustable holders on which said N-1 cameras are mounted, camera synchronization system for taking pictures, a control module for controlling cameras and providing communication between individual parts of the device, configured to issue synchronization commands to the camera synchronization system when capturing a set of images, moreover, the control module is configured to issue commands to the adjustment module to align at least N-1 cameras from among N cameras based on images of the imaginary adjustment target, set shooting parameters for the cameras. 10. The device according to claim 9, in which the control module further performs: color calibration of N-1 cameras to match the colors of the reference images to the colors of the target images. 11. The device according to any one of paragraphs. 9, 10, in which the control module additionally: calculates the coefficients of the necessary polynomial transformations to bring the reference images into correspondence with the target images in spatial coordinates, and applies the polynomial transformations to the said reference images for their more accurate spatial alignment with the target images. 12. The device according to any one of paragraphs. 9-11, in which additionally: Optically coupled cameras with the same lenses and the same radiation detectors form identical optical images of the same scene with good spatial alignment in the central region of the image. 13. The device according to any one of paragraphs. 9-11, in which additionally: Optically coupled cameras with the same lenses and different radiation receivers form optical images of the same scene with different parameters with high spatial alignment when all images are scaled one by one in the central image area. 14. The device according to any one of paragraphs. 9-11, in which additionally: Optically coupled cameras with different lenses and the same radiation receivers form optical images of the same scene with different parameters with high spatial alignment when all images are scaled one by one in the central image area. 15. The device according to any one of paragraphs. 9-11, in which additionally: Optically coupled cameras with different lenses and different radiation receivers form optical images of the same scene with different parameters with high spatial alignment when all images are scaled to one in the central image area. 16. The device according to any one of paragraphs. 9-15, in which at least one optical filter can be inserted in front of at least one camera to independently change the light flux in at least one optical channel of the device. 17. A method for obtaining image sets, comprising the steps of: focus on the required distance N digital cameras, where N≥2, forming a system of digital cameras and requiring focusing, optically coupled using a beam splitter; aligning a system of N digital cameras, where N≥2, optically coupled with N-1 beam splitters, said system of N digital cameras including one or more reference cameras that capture good quality reference images, and one or more target cameras that capture low quality target images using the selected algorithm to achieve good image registration in the central region where optical aberrations are small; using optically coupled digital cameras form optical images of the same scene with good spatial alignment in the central region of the image; receiving a set of N images from a system of N digital cameras synchronized by the start time of shooting with different shooting parameters. 18. The method according to claim 17, in which the synchronization parameters are set in such a way that at least one camera from among the N cameras has a long exposure, and the remaining cameras from among the N cameras have a short exposure, the beginning of which is shifted relative to the start of the long exposure by the required size. 19. The method according to any one of paragraphs. 17, 18, in which: preliminarily registering a set of images of a calibration color target; on the basis of these images, a color correction matrix is calculated, which makes it possible to bring the colors of all images in the set to the colors of one image in the set; based on the calculated color correction matrix, all registered images in the set are color corrected except for one. 20. The method according to any one of paragraphs. 17-19, in which: calculate the coefficients of the necessary polynomial transformations to bring the reference images in the set in line with the target images in spatial coordinates; applying said polynomial transformations to the reference images in the set to more accurately spatially match them to the target images. 21. The method according to any one of paragraphs. 17-19, in which: using optically coupled cameras with the same lenses and the same radiation receivers, optical images of the same scene with identical parameters are formed with good spatial alignment in the central region of the image. 22. The method according to any one of paragraphs. 17-19, in which: using optically coupled cameras with the same lenses and different radiation detectors, optical images of the same scene with different parameters are formed with good spatial alignment when all images are scaled one by one in the central image area. 23. The method according to any one of paragraphs. 17-19, in which: using optically coupled cameras with different lenses and the same radiation receivers, optical images of the same scene with different parameters are formed with good spatial alignment when all images are scaled one by one in the central image area. 24. The method according to any one of paragraphs. 17-19, in which: using optically coupled cameras with different lenses and different radiation receivers, optical images of the same scene with different parameters are formed with good spatial alignment when all images are scaled one by one in the central image area. 25. The method according to any one of paragraphs. 17-23, in which: changing the light flux for at least one of the digital cameras using an optical filter installed in front of at least one of the digital cameras, thereby attenuating or spectral filtering or filtering by polarization of the light flux. 26. Device for obtaining sets of images, containing: two digital cameras, including one reference camera, which captures good quality reference images, and one target camera, which captures poor quality target images, attached to said device and optically coupled with a beam splitter, wherein at least one camera is mounted on automatically adjustable holders, an adjustment module used to align at least one camera based on two images of a scene, the adjustment being carried out by actuating adjustable holders on which said cameras are mounted, digital camera synchronization system for taking pictures, a control module for controlling digital cameras and providing communication between individual parts of the device and obtaining sets of images, wherein the control module instructs the alignment module to align at least one camera based on the images of the scene.

Images