RU2782980C1 - Device and method for image formation with provision of increased depth of the depicted space (options) - Google Patents

Abstract

FIELD: imaging technology.

SUBSTANCE: imaging device providing an increased depth of the depicted space contains an optical unit made with the possibility of simultaneously forming intermediate images of an object at different distances with the effect of blurring at least parts of the images of the object, and containing at least one optical element, and having at least two pupil zones formed in such a way as to provide a predetermined distribution of optical forces and aberrations within each of the pupil zones, on the basis of which the scattering function is formed, characterized by a curve with minimized lateral maxima, while each of the pupil zones corresponds to a given corresponding range of distances to the object and a given corresponding range of angles of the field of view, a receiver made with the possibility of simultaneous registration of intermediate images formed by an optical unit from different distances to the object and at different angles of the field of view, an image processing unit that is performed based on the obtained scattering function in a given range of distances and angles of the field of view, with the restoration at the output of the resulting images without the effect of blurring, regardless of the distance to the object.

EFFECT: provision of an increased depth of the depicted space.

59 cl, 28 dwg

Description

Technical field

The invention relates to methods and devices for forming an image with an increased depth of the depicted space, and can be used, in particular, for photography and video shooting with smartphone cameras, as well as in tablet computers running Android, iOS, HarmonyOS and similar operating systems, as well as in others similar devices in which there is a need to provide an increased depth of the imaged space.

Description of the Prior Art

In the prior art, there are many means of imaging with increased depth of imaging space.

These include the following patent publications:

An imaging system with an optimized increased depth of field is disclosed in US20190212478A1, publ. 07/11/2019, IPC G02B3/00, in which the optical processor contains a pattern in the form of spaced areas with different optical properties. The pattern is configured to form two phase masks in the form of a phase encoder and a dispersion profile encoder. The phase encoder affects the shape of the modulation transfer function (depending on defocusing), increasing the depth of field of the lens block, while the spectral components of the transfer function are shifted relative to each other. The dispersion profile encoder compensates for this shift. The disadvantages of this technical solution is the presence of many optical components that make up the imaging system, namely the imaging lens unit, two phase masks, which affects the quality of the image being formed. At the same time, in the specified device, as clearly illustrated in the graph of Fig. 4B of US20190212478, the resulting transfer function profile is non-uniform, making post-processing of the image difficult, and no image processing means are disclosed to ensure deblurring of the resulting image.

US7593161B2, publ. 09/22/2009, IPC G02B 5/02, discloses an apparatus and method for imaging with increased image depth. The increased depth of field is achieved by an imaging computer system that combines a multifocal imaging subsystem to create a blurry intermediate image with a digital processing subsystem to create a reconstructed image having an increased depth of field. The multifocal imaging subsystem preferably uses spherical aberration as the dominant feature of targeted blur. The central shielding of the multifocal imaging subsystem makes the object's point spread functions more uniform over the range of distances to the object. The iterative digital deconvolution (inverse convolution) algorithm for converting an intermediate image into a reconstructed image contains a metric parameter that increases the efficiency of the algorithm and improves image quality. The disadvantages of this technical solution is the presence of many optical components that make up the multifocal imaging system, which affects the quality of the generated image. At the same time, only spherical aberration is used to increase the depth of the imaged space, and image processing means are not disclosed to ensure the elimination of blurring of the resulting image.

US7616842B2, Nov. 10, 2009, IPC G06K7/00, discloses a method and system for designing an imaging system, including an optical subsystem, a detection subsystem, and a digital imaging subsystem, including at least one digital filter, wherein according to the method, prediction of the characteristics of the final images using a spatial source model and models for the optical subsystem, the detection subsystem, and the digital image processing subsystem. The optical subsystem and digital image processing subsystems are designed together, taking into account the properties of the entire system as a whole. The intermediate image produced by the optical subsystem need not be of high quality, since the quality can be adjusted by the digital image processing subsystem.

Disadvantages: This solution proposes a design method rather than an extended depth of field (EDoF) imaging system.

US11022815B2, publ. 06/01/2021, IPCG02C 7/02, discloses systems and methods for providing improved image quality with increased depth of imaging, which include ophthalmic treatments and ophthalmic lenses such as contact lenses and intraocular lenses (IOLs). The IOL includes first and second lens surfaces. The first or second surface is a refractive surface with an aspherical or diffractive profile. The refractive surface with an aspherical profile can focus light towards the far focus. The diffractive profile surface may include a central region that distributes a first portion of light toward the far focus and a second portion of light toward the intermediate focus. The diffractive profile may also include a peripheral zone surrounding a central zone that distributes a third of the light towards the far focus and a fourth of the light towards the intermediate focus. This solution is aimed at medical use only for contact and intraocular lenses and does not provide for the presence of an image processing tool to eliminate the blurring of the image being formed.

In US5715031A, publ. 02/03/1998, IPC G02C7/04, discloses concentric aspherical multifocal lens designs that use a combination of an aspherical anterior surface that results in reduced aberrations and increased perceived contrast, along with a concentric multifocal posterior surface, to create a lens design that provides clear vision at a long distance, as well as near without loss of contrast. The aspherical surface improves the modulation transfer function (MTF) of the lens-eye system, which improves image focus and contrast, both at far and near distances. This solution is aimed at medical use only for contact and intraocular lenses and does not provide for the presence of an image processing tool to eliminate the blurring of the image being formed.

In US8985767B2, publ. 03/24/2015, IPC G02C 7/02, a method for creating a progressive lens is disclosed, including measuring wavefront distortion by the eye, determining the initial design of a progressive lens based on the wavefront measurement, determining information about how changes in one or more aberrations of the higher order of the eye affect the correction of second order aberrations of the eye based on the information obtained from the measurement of the wavefront. Based on the obtained data, the original progressive lens design is modified to obtain the final progressive lens design. This solution is intended for use in medical purposes for vision correction and does not apply to systems with an increased depth of imaging space.

In the application EP01865827, publ. 12/19/2007, IPC A61B 3/103, describes a method and device for changing the properties of an optical system by means of a continuous multifocal profile, in which the profile contains a component for increasing the depth of field of the image formed by the optical system, while the specified profile contains a component for increasing the depth of field, which is calculated on the basis of the Zernike polynomial of the fourth order. In this case, a change in the basic refraction of the eye is prevented by the fact that the component for increasing the depth of field is additionally calculated based on the Zernike polynomial of the second order. This solution is aimed at medical use only for contact and intraocular lenses and only calculations based on Zernike polynomials of the second and fourth orders are used to increase the depth of field, but it does not provide for an image processing system to eliminate the blurring of the generated image.

JP2011232606, publ. 11/17/2011, IPC G01B 11/24, reveals camera, optical system, birefringent interchangeable lens. In this case, the birefringent device is configured to selectively attenuate the side maxima (side lobes) of the lens scatter function during defocusing. This optical system requires additional optical units, in particular a birefringent device, which significantly increases its cost.

The closest analogue of the claimed invention is an optical system with an increased depth of imaging space, disclosed in US8559118, publ. 10/15/2013, IPC G02B 13/18. Said optical system provides an asymmetric modulation transfer function (MTF). In some aspects, the asymmetric MTF results in an increase in the depth of the depicted space for nearby objects. The optical system consists of a set of lenses (for example, four lenses or five lenses), where the first lens has a positive optical power, and both the lens surface facing the side of the object and the surface facing the image side have convex shapes; a first lens having a positive refractive power, wherein both the surface facing the object side and the surface facing the image side have convex shapes; the second lens has a negative optical power and a meniscus shape, and the surface facing the side of the object has a concave shape at the optical axis, and the surface facing the image side has a convex shape at the optical axis; wherein the fourth lens has a meniscus shape at the optical axis, while the surface facing the object side has a convex shape near the optical axis, and the surface facing the image side has a concave shape near the optical axis. In this case, the length of the optical system remains smaller, approximately 5.3 mm. These optical systems can be used for a high resolution compact camera, for example, in combination with an electronic computing device, a communication device, a display device, surveillance equipment, or the like. In this device, special lens shapes are used, and the authors thereby try to provide the required optical characteristics of the device. However, operating only with the shape of the lenses, it is not possible to obtain invariant optical characteristics depending on the distance to the object. The disadvantage of this optical system is the small depth of the imaged space and the unevenness of the modulation transfer function depending on the defocusing.

Unlike the prototype (US8559118), where a special lens shape is used to obtain increased depth, the claimed imaging device uses a multi-zone optical system, where each zone operates at its own depth. The optical system of the prototype forms a scattering function with side lobes, which leads to image distortions of distant chiseled objects: they are formed with halos in the form of rings or semirings. The multizone optical system of the present invention solves this problem. At the same time, although the presence of an image processing unit is assumed in the prototype, the implementation details and the algorithm of its operation are not disclosed in US8559118. The present inventors use an image processing unit to reconstruct a blurred intermediate image based on a spread function (PSF) or modulation transfer function (MTF) of an optical imaging device, the parameters of the optical system and the image processing unit being jointly calculated using end-to-end optimization techniques.

The objective of the present invention is to develop a method and device for imaging with an increased depth of imaging space.

At the same time, the inventors have developed a method and device that eliminates the disadvantages of traditional imaging methods with the provision of an extended or increased depth of the imaged space:

-providing an extended depth of field (EDoF) of the optical system of a smartphone camera without autofocus function,

-no focus delay

-fast image recovery algorithm,

- no need for user interaction (no need to select a focus area),

- low-cost, compact, optical system of the claimed imaging device, used for mass production.

Thus, the objective of the claimed invention is to create a method and device for forming an image with an increased depth of the imaged space without additional optical elements, which forms high-quality images after processing using artificial neural networks using information about the scattering function.

The main problems of the prior art in the field of smartphone cameras is that:

known autofocus cameras use moving components, such as an electromagnetic drive to move the camera lens relative to the receiver, which in turn causes autofocus delay and complicates the design of the device,

known cameras without autofocus function use additional optical elements, such as phase plates, to provide an increased depth of field, which leads to additional production costs.

It should be noted that prior art autofocus systems have moving mechanical components (motor, cams, gears) to move the lens to focus on the subject, resulting in a focus delay and affecting the quality of shooting fast moving subjects. However, in some autofocus systems (AF systems), user intervention is required to select the focus area manually and adjust the focus position to eliminate possible errors in autofocus mode.

In addition, in well-known AF systems, when pointing at a target object and forming a clear (sharp) image of the object, an unnatural effect of a blurry background image (bokeh) is observed in the form of halos in the form of ovals, rings or semi-rings (English donut-shaped bokeh).

In Fig.1a and 1b, examples of unnatural representation of the background are shown, where in Fig.1a on the left there is a sharp (clear) image of the girl's face in focus and in the background an image of the background in the form of a tree, where the effect of an unnaturally blurred background is observed in the frame. In fig. 1b, the blurred image of the background is shown in an enlarged view: the gaps between the leaves and branches of the tree far from the camera are shown with halos.

Known extended depth of field (EDoF) systems tend to be expensive designs because the extended depth of field function requires additional optical elements or coatings such as phase masks, phase plates, coded aperture masks that introduce phase delay. to ensure even contrast across the required image depth. An image with low contrast is captured by the receiver and further image processing is performed to increase the contrast and enhance the image quality. At the same time, the presence of phase masks, for example, a mask with a fourth-degree parabola profile, a cubic phase mask (Cubic polynominal shape mask), a Toraldo mask, or a coded aperture in EDoF systems significantly increases their cost or worsens light energy characteristics.

Figures 2a, 2b, 2c schematically show images of a scene with targets located at different distances to the smartphone camera (near zone, middle zone - focus area, far zone) formed by known imaging devices.

At the same time, Fig. 2a shows an image of a scene with a conventional lens of the prior art, where the objects in the form of a Christmas tree and a person are blurred because they are out of focus, and the object in the form of a car, which is in focus of the smartphone camera, has a clear (sharp) image .

On fig.2b shows the image of the scene, obtained using a known lens with an increased depth of imaging space (EDoF), which uses a phase mask. At the same time, all objects in the scene are approximately equally blurred, including the car.

On figs shows the image of the scene after the image processing, obtained by the known EDoF lens according to figs in the presence of a phase mask in the optical scheme.

In this case, all objects of the scene, i.e. a tree, a car and a person are depicted equally sharply.

In FIG. 2d are graphs illustrating the amount of modulation gain (T) or image sharpness versus defocus amount (z) for a conventional lens and an EDoF lens. In Fig. 2d, the curve illustrating modulation gain versus defocus for a conventional lens has a sharp peak in the focus area on the scene target, as illustrated in Fig. 2a, and the curve illustrating modulation gain versus defocus for an EDoF lens has uniform, smooth character in the required defocus range (z), which corresponds to a blurred image of all objects in the scene (2b) within the required depth.

On figa-3c presents a general view of the scheme of the claimed technical solution (fig. 3b), proposed by the authors of the present invention, which conventionally shows the scene at the time of taking the image using a smartphone (see figa) and the image of the scene after processing the captured image (fig.3c).

The main solutions of the claimed invention (Key position 1 and Key position 2).

As can be seen in Fig. 3b, the optical imaging device has several zones on the pupil (PZ1, PZ2).

Each zone is characterized by its own focus (F1, F2) and aberration properties.

Each focus corresponds to some distance (depth) in object space. In this case, each zone of the pupil corresponds to a beam of rays L1, L2, for example, the zone PZ1 corresponds to a beam of rays L1 and a focus F1.

The aberration properties determine the image quality according to the image quality criteria for a range of distances.

The receiver registers images from different distances simultaneously. The images on the receiver contain information about objects at different distances. At the same time, the images received at the receiver are blurry, because objects at different distances may be out of focus and overlap each other. To compensate for the blur, a function is provided to process the acquired images by means of an image processing unit (Key position 2), which is capable of reconstructing images at a desired distance (depth) based on the use of information about the characteristics of the optical system (modulation transfer function (MTF) or scattering function in depending on distance). As a result of this image processing on the smartphone screen, a clear image of objects across the entire field of the scene is obtained (see Fig.3c).

The following are the main advantages and characteristics of the claimed invention in comparison with the prior art.

Claimed invention

1)provides an imaging device with increased depth of imaging space (EDoF),

2) used in smartphone cameras and similar devices,

only data from the current frame is required for image reconstruction and information about the characteristics of the system (scattering function or modulation transfer function),

3) the imaging device is an optical unit with a matrix receiver and an image processing unit,

4) The focal length of the imaging device is generally less than 100mm, preferably about 3mm in a smartphone camera,

5) the optical block contains, as a rule, more than two lenses and the formation of an intermediate blurred image on the receiver is provided,

6) a multi-zone optical system is used, where each zone operates at its own depth.

7) the claimed solution provides the formation of an image of an object with high sharpness, regardless of the distance to the object, while there are no boundaries between parts of the image and the resolution of the image over the field of view does not change.

State of the art

A - Known solutions (US11022815B2, US5715031A, US8985767, EP01865827B1 used in ophthalmology):

1) provide vision correction, provide image formation on the retina,

2) applied in contact lenses, IOLs and eyeglasses,

3) contain an imaging device, which is a single lens that works in conjunction with the eye,

4) have a focal length of the imaging device usually more than 150mm,

5) the optical unit contains only one lens and no image processing is provided,

6) provide image formation at far and near distances, which requires accommodation of the user's eye.

B - Known solutions (US20190212478, US7593161, US8559118 used in imaging systems with increased image depth).

1) the optical system has a significant number of optical components (specially shaped lenses, phase masks,) that make up the imaging system, which affects the quality of the image being formed and the cost of the system,

2) the resulting transfer function profile (MTF) is non-uniform, which makes post-image processing difficult,

3) The optical system generates a scattering function (PSF) with side lobes, which leads to image distortions of distant point objects (they are formed with halos in the form of rings),

4) there is no image processing unit or, if it is available, the features of its operation algorithm in conjunction with the optical system are not disclosed.

In doing so, the inventors have developed a method and apparatus for forming an image with an extended or increased depth of the imaged space, eliminating the shortcomings of the prior art solutions described above.

The essence of the invention

According to a first aspect of the invention, there is provided an imaging device for providing increased image depth, comprising:

- an optical unit configured to simultaneously form intermediate images of an object at different distances with the effect of blurring at least parts of the object images, and containing at least one optical element and having at least two pupil zones formed in such a way as to provide a predetermined distribution of optical powers and aberrations within each of at least two pupil zones, on the basis of which a scattering function is formed, characterized by a curve with minimized side maxima compared to the central maximum in a given range of distances to the object, while each of at least two pupil zones correspond to the given corresponding range of distances to the object and the given corresponding range of angles of the field of view,

- a receiver configured to register intermediate images formed by the optical unit from different distances to the object and at different angles of the field of view, simultaneously,

- an image processing unit connected to the receiver and configured to process intermediate images of an object with a blur effect of at least parts of the object images recorded by the receiver, wherein said image is processed based on the obtained scattering function in a given range of distances and angles of the field of view, with restoring the output of the resulting images without the blur effect, regardless of the distances to the object.

At the same time, the minimized side maxima of the scattering function formed by the optical unit do not exceed 10% of the central maximum in a given range of distances to the object.

According to the invention, the predetermined range of distances to the object is from 400mm to 5000mm and the predetermined range of angles of the field of view is from -40 to +40 degrees.

In addition, the optical block is designed in such a way that it provides the formation of an invariant scattering function that does not depend on the distance to the object in a given range, or

independent of the angle of the field of view, or independent of the angle of the field of view and the distance to the object.

The optical unit may be configured to form a uniform modulation transfer function (MTF) independent of the distance to the object.

In addition, the optical unit can be capable of simultaneously forming intermediate images of an object at different distances with the effect of blurring the entire image of the object and contains at least one optical element, which is at least one of: a lens with different surface profiles, a composite lens, a diffractive optical element, holographic optical element, polarizing element, amplitude-phase mask.

One or more lenses of the optical unit may have an aspherical surface and/or, in particular, a Zernike surface, providing aberration correction.

At the same time, according to one variant of the invention, the optical elements of the optical block are located along the optical axis from the side of the object to the image surface: the first lens, the second lens, the third lens, the fourth lens, the fifth lens and the sixth lens, while

the first lens has a positive power, wherein the surface of the first lens on the object side is convex and the surface of the first lens on the side of the image surface is concave;

the second lens has a negative optical power, while the surface of the second lens on the object side is concave and the surface of the second lens on the side of the image surface is also concave;

the third lens has a positive optical power,

the fourth lens has the shape of a meniscus,

the fifth lens and the sixth lens each have a meniscus shape in the region close to the optical axis, while the surface of each of the fifth and sixth lenses, on the object side, is convex in the region close to the optical axis, and the surface of each of the fifth and sixth lenses lens, from the side of the image surface is concave in the area close to the optical axis.

At the same time, in the optical unit, a composite lens is at least one of: a bifocal lens, a progressive lens, a Fresnel lens and / or combinations thereof, a diffractive optical element is a diffractive microstructure configured to provide amplitude-phase modulation of transmitted or reflected radiation , the holographic element is designed and positioned to contribute to the elimination of chromatic aberrations.

In addition, the polarizing element is a polarizer and/or a phase plate and/or a polarizing filter, while the polarizing element, which is a polarizer and/or a phase plate, is configured to ensure multiple passage of radiation through the optical elements of the optical unit, and the polarizing element representing a polarizing filter, designed to eliminate glare.

It should be noted that in the device according to the invention, the amplitude-phase mask is made and placed in the optical block in such a way as to provide a phase delay or amplitude modulation of the radiation to ensure a uniform modulation gain over the required image depth and is one of: masks with a phase profile fourth-degree parabola, cubic phase mask, mask with concentric annular holes in the screen.

At the same time, each of the optical elements of the optical unit is made of an optically transparent material selected from at least one of: optical glass, optical crystal and polymer.

According to the first aspect of the invention, the device may include a shutter located in the plane of the aperture diaphragm or the plane of the imaging device associated with it, and configured to regulate the opening or closing of a given number of pupil zones.

The receiver in the claimed device is a matrix photodetector, in particular a CMOS matrix photodetector or a CCD detector, which is configured to detect electromagnetic radiation in the range of 0.4-0.7 μm, and the sensor pixel size is from 0.7 μm to 1.5 µm.

The image processing unit is configured to process intermediate images of the object with restoring the resulting images at the output based on the convolutional neural network and/or the Wiener filter, the parameters of which are preliminarily determined based on the parameters of the optical unit and the receiver.

At the same time, the parameters of the optical block include at least one of: the radii of the surfaces of the optical elements, the coefficients of the aspherical surfaces of the optical elements, the thicknesses of the optical elements, the refractive indices of the materials of the optical elements, the dispersion value of the optical elements, the distances between the optical elements in the optical block, the length of the device, representing the axial distance between the surface of the first optical element on the side of the object and the image surface on the receiver, and the receiver parameters include at least one of: spectral sensitivity, noise, digitization parameters.

According to a second aspect of the invention, there is provided an imaging device for providing increased image depth, comprising:

- an optical unit configured to simultaneously form intermediate images of an object at different distances with the effect of blurring at least parts of the object images, and containing a set of optical elements, and having at least two pupil zones formed in such a way as to provide a predetermined distribution of optical forces and aberrations within each of at least two zones of the pupil, on the basis of which a scattering function is formed, characterized by a curve with minimized lateral maxima compared to the central maximum in a given range of distances to the object, while each of at least two zones of the pupil corresponds to the given corresponding range of distances to the object and the given corresponding range of angles of the field of view, while the set of optical elements contains, located along the optical axis from the side of the object to the image surface: the first lens, the second lens, three Tew lens, fourth lens, fifth lens and sixth lens, while

the first lens has a positive power, wherein the surface of the first lens on the object side is convex and the surface of the first lens on the side of the image surface is concave;

the second lens has a negative optical power, while the surface of the second lens on the object side is concave and the surface of the second lens on the side of the image surface is also concave;

the third lens has a positive optical power,

the fourth lens has the shape of a meniscus,

the fifth lens and the sixth lens each have a meniscus shape in the region close to the optical axis, while the surface of each of the fifth and sixth lenses, on the object side, is convex in the region close to the optical axis, and the surface of each of the fifth and sixth lenses lens, from the side of the image surface is concave in the area close to the optical axis,

- a receiver configured to register intermediate images formed by the optical unit from different distances to the object and at different angles of the field of view, simultaneously,

- an image processing unit connected to the receiver and configured to process intermediate images of an object with a blur effect of at least parts of the object images recorded by the receiver, wherein said images are processed based on the obtained scattering function in a given range of distances and angles of the field of view, with restoring the output of the resulting images without the blur effect, regardless of the distances to the object.

At the same time, the minimized side maxima of the scattering function formed by the optical unit do not exceed 10% of the central maximum in a given range of distances to the object.

According to the invention, the predetermined range of distances to the object is from 400mm to 5000mm and the predetermined range of angles of the field of view is from -40 to +40 degrees.

In addition, the optical block is designed in such a way that it provides the formation of an invariant scattering function that does not depend on the distance to the object in a given range, or

independent of the angle of the field of view, or independent of the angle of the field of view and the distance to the object.

The optical unit may be configured to form a uniform modulation transfer function (MTF) independent of the distance to the object.

In addition, in the device according to the second aspect of the invention, the optical unit is further configured to simultaneously form intermediate images of the object at different distances with the effect of blurring the entire image of the object.

At the same time, each of the optical elements of the optical unit is made of an optically transparent material selected from at least one of: optical glass, optical crystal and polymer.

According to the second aspect of the invention, the device may include a shutter located in the plane of the aperture diaphragm or the plane of the imaging device associated with it, and configured to regulate the opening or closing of a given number of pupil zones.

The receiver in the claimed device is a matrix photodetector, or a CMOS matrix photodetector or a CCD receiver, which is capable of detecting electromagnetic radiation in the range of 0.4-0.7 μm, and the sensor pixel size is from 0.7 μm to 1.5 μm .

The image processing unit is configured to process intermediate images of the object with restoring the resulting images at the output based on the convolutional neural network and/or the Wiener filter, the parameters of which are preliminarily determined based on the parameters of the optical unit and the receiver.

At the same time, the parameters of the optical block include at least one of: the radii of the surfaces of the optical elements, the coefficients of the aspherical surfaces of the optical elements, the thicknesses of the optical elements, the refractive indices of the materials of the optical elements, the dispersion value of the optical elements, the distances between the optical elements in the optical block, the length of the device, representing the axial distance between the surface of the first optical element on the side of the object and the image surface on the receiver, and the receiver parameters include at least one of: spectral sensitivity, noise, digitization parameters.

According to a third aspect of the invention, there is provided a method for generating an image with increased image depth, comprising the steps of:

- provide simultaneous formation of intermediate images of the object at different distances with the effect of blurring at least parts of the images of the object by means of an optical unit having at least two pupil zones formed in such a way as to provide a predetermined distribution of optical powers and aberrations within each of the at least two pupil zones, on the basis of which a scattering function is formed, characterized by a curve with minimized lateral maxima compared to the central maximum in a given range of distances to the object, while each of at least two pupil zones corresponds to a given corresponding range of distances to the object and specified appropriate range of field of view angles,

- provide registration of intermediate images formed by the optical unit, from different distances to the object and at different angles of the field of view, simultaneously, by means of a receiver,

- the image processing unit processes the intermediate images of the object with the effect of blurring at least parts of the images of the object, based on the obtained scattering function in a given range of distances and angles of the field of view, and

- restore the resulting images without the blur effect, regardless of the distances to the object.

At the same time, the minimized side maxima of the scattering function formed by the optical unit do not exceed 10% of the central maximum in a given range of distances to the object.

According to the invention, the predetermined range of distances to the object is from 400mm to 5000mm and the predetermined range of angles of the field of view is from -40 to +40 degrees.

In addition, the optical block is designed in such a way that it provides the formation of an invariant scattering function that does not depend on the distance to the object in a given range, or

independent of the angle of the field of view, or independent of the angle of the field of view and the distance to the object.

The optical unit may be configured to form a uniform modulation transfer function (MTF) independent of the distance to the object.

In addition, in the device according to the third aspect of the invention, the optical unit is further configured to simultaneously form intermediate images of the object at different distances with the effect of blurring the entire image of the object.

The image processing unit is configured to process intermediate images of the object with restoring the resulting images at the output based on the convolutional neural network and/or the Wiener filter, the parameters of which are preliminarily determined based on the parameters of the optical unit and the receiver.

At the same time, the parameters of the optical block include at least one of: the radii of the surfaces of the optical elements, the coefficients of the aspherical surfaces of the optical elements, the thicknesses of the optical elements, the refractive indices of the materials of the optical elements, the dispersion value of the optical elements, the distances between the optical elements in the optical block, the length of the device, representing the axial distance between the surface of the first optical element on the side of the object and the image surface on the receiver, and the receiver parameters include at least one of: spectral sensitivity, noise, digitization parameters.

Brief description of the drawings

The above and other features and advantages of the present invention are explained in the following description, illustrated by drawings, in which the following is presented:

Fig. 1a is a schematic representation of a generated image of a scene including a target object and a background according to the prior art.

Fig. 1b is a schematic, enlarged view of the background of the generated image of the scene of FIG. 1a according to the prior art.

Fig. 2a is a schematic representation of a scene image formed by a conventional prior art lens.

2b is a schematic representation of a scene image obtained using a known extended depth of field (EDoF) lens.

2c is a schematic representation of the scene image after the image processing process obtained by the known EDoF lens according to fig.2b.

2d are modulation transfer function plots illustrating the modulation gain (T) of an image versus defocus amount (z) for a conventional lens and an EDoF lens.

Fig. 3a-3c is a general view of the diagram of the claimed technical solution, where in Fig.3a is a conditional representation of the scene at the time of taking an image using a smartphone, Fig.3b is a diagram of the claimed technical solution (Key position 1 and Key position 2) and 3c shows the image of the scene after processing the captured image.

Fig. 4 is a schematic representation of an image forming apparatus providing an increased depth of imaging space and the intermediate blurred image generated by it and the output sharp (clear) image according to the invention.

Figs. 5a-5c are schematic representations of the imaging process of the image forming apparatus providing increased imaging depth without the image processing process according to the first embodiment of the invention, where Fig. 5a is a schematic representation of the scene at the moment of capturing an image using a smartphone with EDoF, Fig. .5b shows the imaging process without image processing, fig.5c shows the intermediate blur image being formed.

6a is a plot of the spread function (PSF) for a defocused position autofocus system according to the prior art.

6b is a plot of the spread function (PSF) for an EDoF system according to the prior art.

Fig. 6c is a plot of the spread function (PSF) of an inventive imaging device with increased imaging depth.

Fig. 7 is a plot of scatter functions (PSFs) for an imager providing increased image depth as a function of field of view angle (W) and object distance (d) according to an embodiment of the invention.

Fig. 8 is a diagram of the formation of an intermediate blurred image at the receiver of the image forming apparatus, providing an increased depth of the imaged space and the formation of an output sharp (clear) image after the restoration process by the image processing unit according to an embodiment of the invention.

Fig. 9a is a diagram of intermediate imaging by an imaging device providing an increased depth of imaging space having at least two pupil zones according to an embodiment of the invention.

9b is a modulation transfer function plot showing modulation gain (T) versus defocus (z) for an enhanced depth imaging device with two, three, and multiple pupil zones according to an embodiment of the invention.

Fig. 10 is a diagram of intermediate imaging by an enhanced image depth imaging device having at least two pupil zone subzones according to an embodiment of the invention.

Fig. 11a is a diagram of the formation of two beams of beams with corresponding foci F1, F2 for an imaging device with two pupil zones according to an embodiment of the invention.

Fig. 11b is an enlarged view of the path of the beams of rays in the region of the foci F1, F2 from the diagram of figa according to an embodiment of the invention.

Fig. 11c is a graph illustrating the modulation transfer function (MTF) of modulation gain (T) versus defocus (z) according to an embodiment of the invention.

12a shows the ray path for a dual focus imaging scheme according to an embodiment of the invention.

Fig. 12b shows the ray path for a three focus imaging scheme according to an embodiment of the invention.

12c is a graph illustrating the modulation transfer function (MTF) of modulation gain (T) versus defocus (z) for the dual focus imaging circuit of FIG. 12a.

Fig. 12d is a graph illustrating the modulation transfer function (MTF) of modulation gain (T) versus defocusing (z) for the three focus imaging scheme of Fig. 12b.

Fig. 13 is a schematic representation of the optical block of the image forming device providing an increased depth of the imaged space with the division of the pupil zone into at least two zones, where the scattering function (PSF) does not depend on the distance to the object, while scattering functions are presented under position 13a) (PSF) for the far position, center position and near position, and under the position 13b) are the scattering functions (PSF) depending on the angle of the field of view.

Fig. 14 is a schematic representation of the optical unit of the imaging device with increased depth of image space with the division of the pupil zone into at least two zones, where the scattering function PSF depends on the depth of field of view, while under the position 14a) scattering functions (PSF) are presented depending on the depth of the imaged space for the far position, the central position and the near position, and under the position 14b) the PSF functions are presented depending on the angle of the field of view.

Fig. 15a is a schematic view of scatter spots in the image plane in accordance with the scatter function (PSF) for an imaging device providing an increased imaging depth according to Fig. 13.

Fig. 15b is a schematic view of scatter spots in the image plane according to the scatter function (PSF) for an imaging device providing an increased imaging depth according to Fig. 14.

15c is a graph illustrating the modulation transfer function (MTF) of modulation gain (T) versus defocus (z) according to embodiments of the invention.

Fig. 16a is a schematic representation of the device for forming the invention providing an increased depth of imaging space with a shutter without an image processing device.

16b, 16c, 16d are graphs illustrating modulation transfer functions (MTFs) characterizing modulation gain (T) versus defocus (z) versus pupil opening according to embodiments of the invention.

Fig.17a, 17b,17c,17d - maps of the distribution of optical power depending on the shape of the pupillary areas.

Fig. 18 is a schematic representation of an enhanced image depth imaging apparatus and its intermediate image and output image according to the invention.

19a and 19b show images obtained by the inventive imaging device with increased image depth before and after processing by the imaging device.

Fig. 20a is a diagram of an exemplary implementation of an optical unit of an image forming apparatus providing an increased imaging depth according to the invention.

20b is a graph illustrating the modulation transfer function (MTF) of the prior art modulation gain (T) versus defocus (z).

20c is a graph illustrating a modulation transfer function (MTF) characterizing the modulation gain (T) versus defocus (z) of an inventive imaging device to provide increased image depth.

Fig. 21 - graphs illustrating the modulation transfer functions (MTF) characterizing the dependence of the modulation transfer coefficient (T) on defocusing (z) for an optical unit having an aspherical surface and / or a Zernike surface according to an embodiment of the claimed imaging device with increased image depth space.

22a shows examples of MTF transfer function plots for an optical unit of an imaging apparatus according to the invention.

Fig. 22b shows examples of PSF plots for an optical unit of an imaging apparatus versus object distance (d) according to an embodiment of the invention.

Fig. 22c is a graph illustrating the distortion of an optical unit of an image forming apparatus with increased image depth according to an embodiment of the invention.

Fig.23a - schematically shows the optical unit with the division of the entrance pupil into two zones of the pupil (PZ1, PZ2) and the receiver according to an embodiment of the imaging device with an increased depth of imaging space.

Fig.23b - shows the profile of the distribution of the optical power of the optical unit in the sagittal section of the pupil according to Fig.23a imaging device with increased depth of imaging space according to the invention.

Fig. 23c shows the profile of the distribution of the focal length of the optical unit in the sagittal section of the pupil according to Fig. 23a of the imaging device providing an increased depth of imaging space according to the invention.

Fig. 23d is a map of the distribution of the focal length of the optical unit according to Fig. 23a of the image forming apparatus providing an increased depth of imaging space according to the invention.

24a is a plot of the scatter function (PSF) with minimized side lobes of an optical unit of an imaging device to provide increased image depth according to the invention.

24b is a plot of the Modulation Transfer Function (MTF) versus defocus (depth of field) illustrating the smooth curve of the MTF Function.

Fig.25a-25e - the process of image formation by the optical unit and registration on the receiver, where Fig.25a shows the conditional division of the object into parts, Fig.25b shows a mathematical representation of the image formed by the optical unit in the form of a convolution of the intensity distribution and scattering function , 25c shows the calculation of the scattering function for each part of the receiver, 25d shows the image acquisition operations at the receiver, 25e shows the acquisition of an intermediate image at the Yc receiver.

Fig. 26 is a schematic view of an object image (Yc) obtained at the receiver (see resulting intermediate image in Fig. 25d) and a reconstructed image obtained after processing in the image processing unit.

27 is a schematic representation of an example of a shared spread function (PSF) according to an embodiment of the invention.

28 is a schematic representation of an example of a seven layer neural network architecture according to an embodiment of the invention.

Preferred embodiments of the invention

Exemplary embodiments of the present disclosure will be described in detail below. Exemplary embodiments have been illustrated in the accompanying drawings, in which the same or like reference numerals refer to the same or like elements or elements having the same or similar functions. The exemplary embodiments described with reference to the accompanying drawings are illustrative, used only to explain the present disclosure, and should not be construed as limiting it in any way.

Within the framework of the present invention, the following concepts and terms will be used, the interpretation of which is provided below by the authors of the invention:

Target object - the object selected by the user for shooting.

Aperture diaphragm - a diaphragm that limits the beam of rays emerging from the axial point of the object.

The pupil is a paraxial image of the aperture diaphragm.

Pupil zone (PZ) - a selected area of the pupil with its own properties of the optical system (focal length, aberration properties).

The field of view is the angle between two rays passing through the center of the entrance pupil of the lens (lens) to the displayed points of the object in the space of objects that are farthest from the optical axis.

Field of view angle - the angle between the optical axis and the beam passing through the point of the object in the field of view

A scatter spot is an image of a point formed by a real optical system.

An intermediate image is an image formed by the optical unit at the receiver.

The reconstructed image is the image at the output of the image processing unit.

Point Spread Function (PSF) - describes the dependence of the illumination distribution of the image of a point object on the coordinates in the image plane.

Bokeh effect (boken-effect) - blur (unsharpness) of the part of the image, usually the background, that is out of focus.

Halo - a light border around the image of the points of the subject.

The modulation transfer coefficient is the ratio of the image contrast of a harmonic (sinusoidal) object to the contrast of the object itself.

Modulation transfer function (MTF) - shows the change in the modulation transfer coefficient (contrast) of the image depending on the spatial frequency of the harmonic object. In other words, it characterizes how accurately the spatial-frequency content of the object is transmitted to the image.

Defocusing - displacement of the analysis plane relative to the receiver installation plane.

Optical system (unit) - a set of optical elements located in space in a certain way, providing the necessary formation of beams of rays (without a receiver and a processing unit).

According to the first implementation variant, an imaging device is provided to provide an increased depth of the imaged space (see Fig. 4), containing an optical unit, a receiver and an image processing unit. Figures 5a-5c show a diagram of an image forming apparatus providing an increased depth of imaging space without an image processing unit. While the optical unit has at least two zones of the pupil (PZ1, PZ2) (see Fig.3b,4 and 5b). In this case, each pupil zone with the corresponding aberration properties operates with the appropriate range of distances (d1, d2, d3) to the object, which corresponds to its focus (F1, F2, F3) and the range of field of view angles (angle 1, angle 2, angle 3) . In this case, the aberration properties (for example, wave aberrations (OPD)) of each pupil zone determine the image quality in accordance with the characteristics of the optical block for the distance range d1, d2, d3, ..., dN. Characteristics include, for example, the spread function (PSF) or modulation transfer function (MTF) for a range of object distances and field of view angles. In this case, the optical unit forms an intermediate blurred image, which is further corrected by the image processing unit, using the characteristics of the optical system (PSF or MTF functions) in the range of distances to the object and angles of the field of view, and at the output of the image processing unit, a clear contrast image of objects is provided throughout the field of view .

The characteristics and properties of the pupil areas are jointly optimized so that the optical unit generates a spread function (PSF) with minimized side lobes (side maxima, side peaks) for out-of-focus points.

Further on figa-6c presents examples of graphs of the scattering function (PSF). Figure 6a is a graph of the scattering function (PSF) when the optical system of the camera is defocused according to the prior art, as a result, the images of point objects of the background are scattered spots in the form of rings or semi-rings, which leads to an unnatural blurring of the background (donut-shaped bokeh).

Fig. 6b is a plot of the defocus scatter function (PSF) for a prior art EDoF system, the plot shows side maxima (side lobes), resulting in background point object images being scatter spots with a ring-shaped halo, resulting in unnatural background blur (donut-shaped bokeh).

6c is a plot of the spread function (PSF) of an inventive imaging device in which the PSF is in an out-of-focus position without side lobes or with side lobes minimized as much as possible, resulting in a uniform background blur.

This effect is achieved by using a multifocal optical unit with multiple pupil zones. At the same time, as a result, at the output of the image processing unit, a natural background blur (bokeh effect) is obtained in the final image of the object.

It should be noted that known EDoF systems work as follows:

1) the optical unit registers a blurry image. It is deliberately made blurry (using amplitude-phase masks or maintaining spherical aberration) in order to increase the depth of the depicted space. For such systems, the scattering function is obtained with side maxima (side lobes). These side maxima lead to the fact that a halo appears in the image of a point in the form of rings or semirings. This is especially critical for distant objects, where there are many point sources, for example, gaps between leaves and tree branches (see figa, 1b). From a distance, they appear to be points, but EDoF registers them as haloed spots due to side peaks in the scatter function.

2) the blurry image formed by the optical unit is recorded by the receiver

3) the processing unit restores a sharp image from a blurry one. If there are blurry points (spots) with halos in the image, then after processing they will be converted into sharp points with a sharp halo. Those. the image is distorted: they were supposed to get a point, but they got a point with a sharp ring.

In the claimed image forming device in the optical unit, blurring is achieved not by a phase mask or by introducing spherical aberration, but by dividing the pupil into zones with their own aberration properties, but the use of a mask is not excluded. As a result, the optical unit generates a blurry image and additionally, unlike other EDoFs known from the prior art, the scattering function (PSF) has no side maxima. A point object (or a point on an object, a point source) is converted to a blurry point (spot), but without ghosting.

In this case, the image processing unit is configured to process the intermediate blurred image (see 4, 5b) and restore the blurred image generated by the optical device. In this case, image processing is performed based on the distribution of the PSF or MTF function over the range of distances to the object and angles of the field of view. In this case, image restoration is also used taking into account the parameters of the image forming device receiver, which include: spectral sensitivity, noise, sampling parameters, and which are used when performing the image restoration process. Thus, the processing unit restores a sharp image from a blurry one. Since there are no ghosting in the image, the blurred image of the dot (spot) turns into a dot, as it should be in a natural-looking image.

Figure 7 presents graphs illustrating Examples of changes in the shape of the scattering function depending on the angle of the field of view and the distance to the subject. In doing so, the inventors clearly demonstrated how the scattering function (PSF) depends on the angle of the field of view (W) and the distance to the object (d). On the graphs in Fig.7, (where x is the coordinate in the image plane, I is the value of the scattering function), examples of the scattering function for the distance to the object (target object) equal to 400mm, 1000mm, 5000mm and field of view angles (W) equal to 0 degrees, 23 degrees and 40 degrees. The information obtained as a result of experimental studies carried out by the inventors, some of which are shown in Fig.7, as a non-limiting example, is then used in the image processing unit to restore the blurred image.

It should be noted that during experimental studies, the inventors found that at each of the distances of 400 mm, 1000 m and 5000 mm to the object, the imaging device according to the invention generates a scattering function with minimized side lobes, which indicates that a uniformly blurred intermediate image is obtained, and that in turn the queue ensures the restoration of a clear (sharp) image after processing in the image processing unit.

In addition, during experimental studies, the inventors found that at angles of view equal to: -40 degrees, 0 degrees, 23 and +40 degrees and at given distances to the object, the imaging device according to the invention generates a scattering function with minimized side lobes, which indicates the receipt of a uniformly blurred intermediate image, and which in turn ensures the restoration of a clear (sharp) image after processing in the image processing unit.

Based on the research, it was found that the imaging device according to the invention provides the formation of a scattering function, in which the minimized side maxima (side lobes) of the scattering function formed by the optical unit do not exceed 10% of the central maximum in a given range of distances to the object.

Fig. 8 is a schematic representation of the blurring process at the receiver, and the image data from the receiver is fed to the image processing unit where the image is reconstructed. Here, in this case, image processing is performed based on the distribution of the spreading function (PSF) as shown in FIG. 7 over a range of object distances and field of view angles. In this case, image restoration is also used taking into account the parameters of the image forming device receiver, which include: spectral sensitivity, noise, sampling parameters, and which are used when performing the image restoration process.

It should be noted that the receiver is a matrix photodetector, in particular a CMOS matrix photodetector or a CCD detector, which is configured to detect electromagnetic radiation in the range of 0.4-0.7 μm, and the pixel size of the receiver is as 0.7 μm , 1 µm, and 1.5 µm. According to another embodiment of the invention, an image forming device is provided, having at least 2 pupil zones, the scheme shown in Fig. 9a.

In this case, each zone of the pupil functions with the corresponding range of distances (d1, d2, dN) to the object (not shown in Fig. 9a), which corresponds to its focus (F1, F2, FN) and the range of the corresponding angles of the field of view. In Fig.9a, each zone of the pupil corresponds to its own beam of rays, i.e. the zone PZ1 corresponds to the beam L1, etc., and its own region of the scattering spot.

In this case, the wave aberrations (OPD) of each pupil area determine the image quality in accordance with the scattering function (PSF) or modulation transfer function (MTF) for a range of distances to the object and angles of the field of view. In this case, image processing is performed based on the dependence of the PSF or MTF function on the range of distances to the object and the angles of the field of view. In this case, image restoration is also used taking into account the parameters of the imaging system receiver, which include: spectral sensitivity, noise, sampling parameters, and which are used when performing the image restoration process.

At the same time on fig.9b presents graphs of modulation transfer coefficient (T) from defocusing (z) for a device with two, three and multiple zones of the pupils, and the corresponding number of foci. The concept of a set in the context of focuses is used conditionally and provides for a range of focuses, for example, from 3.1 mm to 2.9 mm, used in smartphone cameras.

Here, the graph for an imaging device with a plurality of pupil zones, i. e. when the pupil is divided into multiple zones, it represents a smoother dependence of the modulation transfer coefficient (T) on defocusing (z), in contrast to the graphs when the pupil is divided into two or three zones.

Thus, an optical imaging device having more than 2 or 3 pupillary areas is characterized by a smoother modulation transfer function and a greater imaging depth.

According to another embodiment of the imaging device, a structure is proposed in which, in addition to dividing the pupil into 2 zones, each zone is divided into subzones (subzone 1, subzone 2, subzone 3 and subzone N), see Fig.10.

Each subzone has its own focus (F1…FN) and its own beam beam L1, L2, LN. In this case, section 1 is a section of the lens, for example, meridional, and section 2 is sagittal, or vice versa. For an ordinary axisymmetric surface (for example, a spherical one), these sections have the same profile. In the example shown in Fig. 12, section profiles 1 and 2 differ from each other, in this case, the asymmetric lens profile used in the imaging device according to Fig. 10 provides a greater depth of image space and the formation of a uniformly blurred intermediate image throughout the depth and field scenes.

In this case, the optical device generates an intermediate blurred image image, which is further corrected by the image processing unit in accordance with the scattering function (PSF) or modulation transfer function (MTF) for the range of distances to the object and angles of the field of view. In this case, image restoration is also used taking into account the parameters of the imaging system receiver, which include: spectral sensitivity, noise, sampling parameters, and which are used when performing the image restoration process.

It should be noted that depending on the number of pupil zones, each of which has its own focal length (F1, F2, FN), for the corresponding beam of rays (L1, L2, LN) its own modulation transfer function (MTF) is formed.

In FIG. 11a and 11b are diagrams for generating two beams of beams with respective foci F1, F2 for an imaging device with two pupil zones. On figa shows a diagram of the formation of two beams of radiation, converging into foci F1, F2. On fig.11b presents an enlarged view of the path of the rays near the foci F1, F2 (ie in the area, which is highlighted by an oval in fig.11a).

At the same time, such a design with two pupil zones is characterized by a modulation transfer function (MTF), which characterizes the dependence of the modulation transfer coefficient (T) on defocusing (z) (see Fig. 11c), and having two maxima of the same height on the graph, which ensures the formation a clear, sharp image of the same modulation gain at peak points.

On figa and 12b shows the course of beams of rays for schemes with two and three foci, respectively.

12c and 12d show the respective MTF functions as a function of the amount of defocus (z).

In this case, for an optical scheme with two pupillary zones, the optical imaging device provides images of different contrast depending on the different depth (defocusing), see Fig.12c, which clearly illustrates the presence of at least two maxima of different heights. And Fig. 12d shows the MTF function for an optical imaging device with three or more foci, provided by dividing the pupil area into a plurality of zones or sub-zones, which makes it possible to obtain images at different distances to the object without focusing. At the same time, as clearly illustrated in Fig.12d, the presence of many maxima on the graph of the MTF function provides a more uniform character of the MTF function.

According to one embodiment of the imaging device (see Fig.13) with the division of the pupil zone into at least two zones, an optical unit is proposed that illustrates the scheme for the formation of radiation beams L1, L2 with the corresponding foci, where the scattering function PSF does not depend on the distance to object or i.e. depth of the depicted space. Each zone (PZ1, PZ2) is characterized by its own focus (F1, F2) and aberration properties.

It should be noted that the depth of the imaged space and the depth of field - both of these concepts refer to the distance along the optical axis, within which the object plane and the image plane are allowed to move, respectively, provided that the image quality is still satisfactory, i.e. in our case, it can be restored by the processing unit.

In this case, each zone of the pupil corresponds to the corresponding beam of radiation L1, L2, for example, the zone PZ1 corresponds to the beam L1 and the focus F1.

The invariance of the optical block according to Fig. 13 is illustrated by the spread function plots for the far position, central position and near position, which have approximately the same shape of the PSF function (see graphs a) of Fig. 13), regardless of the distance to the object, where PSF2, PSF0 , PSF3 PSFs for far position, center position, and near position, respectively.

Thus, the scattering functions are invariant (PSF0~PSF2~PSF3) regardless of the distance from the object.

At the same time, as can be clearly seen in the graphs b) of Fig.13, the PSF functions differ depending on the angle of the field of view. Thus, the scattering functions are different (PSF0≠PSF1) for different angles of the field of view.

As can be seen in Fig. 15a, which is a view of the stray spots in the image plane according to the PSF function for the imaging apparatus of Fig. 13, i. e. scattering spots differ from each other depending on the angle of the field of view, but are the same in depth.

This capability of the optical unit of the imaging apparatus according to the invention, i. e. the independence of the quality of the formed image from the distance (depth of field) to the target object makes the processing of the obtained images from different distances less complicated.

According to another embodiment of the imaging device (see Fig.14) with the division of the pupil zone into at least two zones, an optical block is proposed illustrating the scheme for forming beams of beams L1, L2 with corresponding foci, where the scattering function PSF depends on the depth of field vision. Each zone (PZ1, PZ2) of the pupil is characterized by its own focus (F1, F2) and aberration properties.

In this case, each zone of the pupil corresponds to the corresponding beam of rays L1, L2, for example, the zone PZ1 corresponds to the beam L1 and the focus F1.

The functionality of the optical unit according to Fig. 14 is illustrated in the spread function plots for the far position, center position and near position, which have a different shape of the PSF function (see graphs a) of Fig. 14) depending on the depth of the imaged space, where PSF2, PSF0, PSF3 PSFs for far position, center position, and near position, respectively. Thus, the scattering functions are different (PSF0≠PSF2≠PSF3) for different depths of the imaged space.

At the same time, as can be clearly seen in the graphs b) of Fig. 14, illustrating the dependence of the PSF function on the angle of the field of view, where the PSF functions are approximately the same depending on the angle of the field of view. Thus, the scattering functions are invariant (PSF0~PSF1) for different angles of the field of view.

As can be seen in Fig. 15b, which is a view of the stray spots in the image plane according to the PSF for the imaging apparatus of Fig. 14, i. e. stray spots have the same size depending on the angle of the field of view, but differ in depth.

This capability of the optical unit of the imaging apparatus according to the invention, i. e. independence of the quality of the generated image from the field of view, provides less complex processing of the resulting images.

According to another embodiment of the imaging apparatus, the features shown in FIG. 13 and FIG. 14 can be combined in one device, i.e. at the same time, the invariance of the scattering function is ensured both in the field of view and in depth, which simplifies the subsequent image processing.

According to another embodiment, the optical unit of the imaging apparatus also provides a uniform modulation transfer function (MTF) characterizing the dependence of the modulation gain on defocus (z), i.e. The MTF is independent of the amount of defocus within the desired range (uniform MTF range in Fig. 15c).

According to another embodiment, the optical unit of the imaging device provides both a uniform modulation transfer function and an invariant scattering function (see Fig. 15c).

This ability of the optical unit of the imaging device according to the invention allows less complex processing of the resulting images.

The optical unit of the image forming device is a set of optical elements located in space in a certain way, providing the necessary formation of beams of rays (without a receiver and a processing unit). In this case, lenses having a different surface profile and/or a composite lens and/or a diffractive optical element and/or a holographic optical element and/or a polarizing element and/or an amplitude-phase mask can be used as optical elements.

In this case, the composite lens is a structure consisting of at least two zones with different optical properties. As a compound lens, a bifocal lens, a progressive lens, a Fresnel lens can be used.

A bifocal lens is a compound lens with two zones having different focal lengths.

A progressive lens is a composite lens consisting of several zones characterized by a focal length gradient.

The Fresnel lens is a composite lens consisting of several stepped zones, which are the corresponding zones of the continuous profile of the surface of a conventional lens.

In this case, the diffractive optical element is a diffractive microstructure that performs amplitude-phase modulation of transmitted or reflected radiation.

A holographic optical element is a diffractive optical element manufactured by the methods of interference of light waves.

The diffractive optical element, and in particular the holographic optical element, makes it possible to effectively correct or introduce chromatic aberrations, having a small thickness of 5 microns.

The polarizing element is a device that changes the state of polarization. For example, a polarizer or a phase plate can be used as the polarizing element. The polarizer is designed to produce polarized or partially polarized radiation. The phase plate introduces the phase difference between the orthogonal linearly polarized radiation components. In particular, a quarter-wave plate can be used as a phase plastic if the phase difference is a quarter of a wavelength, or a half-wave plate if the phase difference corresponds to half a wavelength. The phase plate can be configured to convert radiation from the p-polarization state to the s-polarization state or vice versa, convert the right circular polarization to the left circular polarization or vice versa, convert linearly polarized radiation to circularly polarized or vice versa.

The presence of a polarization element in the optical unit according to the invention makes it possible, for example, to ensure that the radiation passes through a small number of optical elements constituting the optical unit multiple times. At the same time, the multipass passage of radiation makes it possible to reduce the overall length of the optical unit and the image forming device as a whole, thereby ensuring the compactness of the device, which is an important requirement for smartphones.

The polarizing element can also be used as a polarizing filter, for example to eliminate glare.

Amplitude-phase mask - a device that performs amplitude-phase modulation of transmitted or reflected radiation.

Amplitude-phase masks are, for example: masks with a fourth-degree parabola phase profile, a cubic phase mask, masks with concentric annular holes in the screen and are designed to provide a phase delay or amplitude modulation to ensure a uniform modulation gain over the required image depth. The presence of amplitude-phase masks in the optical block provides optimization of the shape of the scattering function (PSF).

At the same time, each optical element from the specified block of optical elements is made of an optically transparent material selected from one of optical glasses, optical crystals and polymers.

In one of the embodiments of the invention, the optical unit provides for the presence of a shutter, for example an optical shutter, preferably in the plane of the aperture diaphragm or the plane of the imaging device associated with it. The presence of an optical shutter allows you to adjust the number of focal lengths of the device, i.e. regulation of the opening or closing of a certain number of pupil zones (PZ). According to Fig. 16a, three zones are provided: PZ1, PZ2 or PZ3, in which case an intermediate image will be obtained at the receiver with the following MTF characteristics, i.e. contrast (T) versus defocus (z) (see graphs in Figs. 16b-16d).

Graph 16b corresponds to the option when all three zones of the pupil are open, option 16c - one zone of the pupil is open and option 16d - PZ2 and PZ3 are open.

The optical shutter allows you to vary the focal length of the system without changing the position of the lenses or changing the focal length of individual lenses.

The presence of an optical shutter in the optical unit entails a decrease in the area of the aperture diaphragm and loss of energy (luminosity) of the optical unit, which is further compensated by image processing in the image processing unit.

According to the invention, the pupil zones are divided into zones and subzones, while the pupil zones can have a different shape and a different distribution of optical powers within the zone.

At the same time, Figs. 17a-17d show maps of the distribution of optical power depending on the shape of the optical zones.

On figa presents a continuous multifocal asymmetric distribution of optical power, through which it is possible to provide a given distribution of the scattering function (PSF) or modulation transfer function (MTF). Such a configuration is quite simple to manufacture.

On fig.17b presents a continuous multifocal axisymmetric distribution of optical power. Such a configuration is quite simple to manufacture.

Figure 17c shows discrete distributions of optical power (P1, P2, PN) axisymmetric with an annular pupil shape and planar-symmetrical with a segmented pupil shape, through which it is possible to provide depth separation of images formed by different zones, and such a power profile simplifies the image processing process .

FIG. 17d shows a discrete multifocal (P1, P2, PN) arbitrary distribution of optical power, by means of which it is possible to provide a given spreading function (PSF) or modulation transfer function (MTF) distribution.

According to one of the embodiments of the invention, a device and a method for forming an image providing an increased depth of the imaged space are provided, based on the joint design of the optical block and the image processing block, while both blocks are calculated and adjusted together, taking into account the features and parameters of each block (the so-called end-to-end optimization ).

Schematically, the image forming device is shown in Fig. 18, where an optical unit is sequentially located, containing a set of optical elements in the form of lenses with different surface profiles through which the beams L1, L2, and L3 pass, a receiver and an image processing unit.

The parameters of the optical block include: parameters of the surfaces of optical elements, for example, the radii of the surfaces of the lenses (r ₁ ,r ₂ ,..,r _i ), parameters of optical materials, for example the refractive indices of optical elements (n ₁ , n ₂ ,.. n _i ) , the thickness of the optical elements and the distance between them in the optical unit (d ₁ ,d ₂ ,..d _i ), L is the length of the system (axial distance between the surface of the first optical element on the side of the object and the image surface on the receiver).

Receiver parameters include, for example, spectral sensitivity κ _λ , noise ƞ, digitization parameters S.

The parameters of the image processing unit include, for example, the parameters of digital filters or the weight coefficients of the layers (w ₁ ,w ₂ , ..., _wi ) of the neural network. The image processing unit, taking into account the above parameters of the optical unit, including the modulation transfer function (MTF) or scattering function (PSF), depending on the distance to the target object and the parameters of the receiver, provides image processing with its subsequent restoration.

и исходным изображением I_λ целевого объекта минимизируются, Taking into account the specified parameters of the optical unit, the receiver and the image processing unit, the image forming device provides the image formation and processing process in such a way that the discrepancy between the reconstructed image

and the source image I _λ of the target object are minimized,

. For example, quadratic discrepancy (1):

.

Thus, the intermediate image formed at the receiver by the optical unit does not have to be of high quality and resolution, which simplifies the requirements for optics, since the image quality is corrected in the image processing unit.

и исходным изображением I_λ (референсное изображение, соответствующее идеальному изображению целевого объекта), нормы их визуального сходства VGG (детальное описание VGG раскрыто в источнике: https://arxiv.org/abs/1409.1556v6) или их комбинации минимизируется. Image processing and restoration is performed in the image processing block in such a way that the evaluation function in the form of the norm of the residual between the restored image

and the original image I _λ (reference image corresponding to the ideal image of the target object), their visual similarity norm VGG (a detailed description of VGG is disclosed in the source: https://arxiv.org/abs/1409.1556v6), or a combination of both is minimized.

As a non-limiting example, the processing block may be implemented using artificial neural networks, a Wiener filter, or a combination thereof.

As one of the solutions, an implementation based on a U-net type network widely used in the field of image processing is proposed, disclosed in the source: (https://arxiv.org/pdf/1505.04597.pdf).

In developing and analyzing the present invention, the authors use the implementation of the above seven-layer network architecture, see FIG. 28. In the downsizing (input image) direction, the network contains blocks of multiple convolutions (see FIG. 28, Block Type1 - Block Type3), and in the upsizing direction (Block Type4 - Block Type7) are used. A detailed description of the blocks is provided in the source: https://arxiv.org/pdf/1505.04597.pdf.

A set of pairs of images (reference image and processed image) is used to train the network. In the process of training, the parameters of the blocks are determined, i.e. weight coefficients of the layers w ₁ ,w ₂ , …, _wi of the neural network, mainly using gradient optimization methods, examples of which are disclosed in: https://arxiv.org/pdf/1609.04747.pdf .

As a basic such set of pairs of images, high-quality images are used for reference images, for processed images, an image model on the receiver. The simplest model describing the influence of an optical system is convolution with the corresponding scattering function PSF with the addition of parameterized noise.

Taking noise into account when building an image at the receiver allows using the network to suppress noise during image processing, which is important in view of the inevitability of the presence of noise and its influence on the processing result.

The scoring function is the L1 norm (the sum of the modules of the pixel difference between the restored and the reference image), or a weighted combination of the L1 norm and the similarity visual perception metric (VGG) disclosed in: https://arxiv.org/abs/1409.1556v6) or their combinations. It is also possible to use alternative solutions based on artificial neural networks, using various network architectures and evaluation functions. For example, the quadratic norm, the inverse evaluation function, or the pyramidal loss function.

It should be noted that the processing of intermediate images of an object based on a convolutional neural network, for example, a U-net type network, is performed taking into account the parameters of the optical unit and the receiver. At the same time, the parameters of the optical block include at least one of: the radii of the surfaces of the optical elements, the coefficients of the aspherical surfaces of the optical elements, the thicknesses of the optical elements, the refractive indices of the materials of the optical elements, the dispersion value of the optical elements, the distances between the optical elements in the optical block, the length of the device, representing the axial distance between the surface of the first optical element on the side of the object and the image surface on the receiver, and the receiver parameters include at least one of: spectral sensitivity, noise, digitization parameters.

One of the alternatives to the solution based on a neural network for restoring the image formed on the receiver is the use of classical algorithmic approaches known from the prior art.

Such, for example, is the method of restoring a blurred image using a Wiener filter (Wiener, Norbert (1949). Extrapolation, Interpolation, and Smoothing of Stationary Time Series. New York: Wiley. ISBN 978-0-262-73005-1.), so-called inverse convolution with Wiener filtering).

Within the framework of this approach, the optimal estimate of a sharp image x from the image formed at the receiver y is obtained by applying the Wiener filter (X, Y are the corresponding images in frequency representation):

, (2)

, (2)

Where

- обратное отношение сигнал-шум устройства формирования изображения,H is the corresponding spread function in frequency representation (i.e. modulation transfer function), and

- inverse signal-to-noise ratio of the imaging device,

X - spectrum of the image in the frequency representation after processing in the image processing unit,

Y is the spectrum of the intermediate image recorded at the receiver.

Solutions based on Wiener filtering make it possible to implement the operation of the image processing unit in real time, and thus, such a solution can be used to implement the function of previewing the reconstructed image by the user.

It should be noted that the processing of intermediate images of the object based on the Wiener filter is performed taking into account the parameters of the optical unit and the receiver. At the same time, the parameters of the optical block include at least one of: the radii of the surfaces of the optical elements, the coefficients of the aspherical surfaces of the optical elements, the thicknesses of the optical elements, the refractive indices of the materials of the optical elements, the dispersion value of the optical elements, the distances between the optical elements in the optical block, the length of the device, representing the axial distance between the surface of the first optical element on the side of the object and the image surface on the receiver, and the receiver parameters include at least one of: spectral sensitivity, noise, digitization parameters.

Acceleration of image processing when restoring the image formed on the receiver is also implemented within the framework of the present invention, if the Scattering Function PSF can be represented in a separable (separable) form, i.e. if the given function can be divided into two one-dimensional signals, in vertical projection and in horizontal projection.

FIG. 27 shows an example of a shared spreading function (PSF) as a horizontal and vertical profile.

The value of each pixel x[r, c] in the image is equal to the corresponding point in the horizontal projection horz[c] multiplied by the corresponding point.

The use of Shared Scattering Functions (PSFs) speeds up the processing of image data.

Figures 19a and 19b show images taken by the inventive imaging device before and after processing by the imaging device.

Figure 19a shows two images: a car and a garage door in the background, at a medium distance, the first (blurred) as the input image captured by the camera, and the second (sharp) as the output image, i. reconstructed image after processing by the image processing unit.

In FIG. 19b shows two images: a balcony and in the background elements of a house, in the far distance, the first as an input image captured by the camera, and the second (sharp) as an output image, i.e. reconstructed image after processing by the image processing unit.

Further on figa shows an example of the implementation of the optical unit of the image forming device, which contains optical components sequentially located along the optical axis from the side of the object to the image surface: aperture diaphragm, the first lens (l ₁ ), the second lens (l ₂ ), the third lens (l ₃ ), the fourth lens (l ₄ ), the fifth lens (l ₅ ) and the sixth lens (l ₆ ), while

the first lens has a positive power, wherein the surface of the first lens on the object side is convex and the surface of the first lens on the side of the image surface is concave;

the second lens has a negative optical power, while the surface of the second lens on the object side is concave and the surface of the second lens on the side of the image surface is also concave;

the third lens has a positive optical power,

the fourth lens has the shape of a meniscus,

the fifth lens and the sixth lens each have a meniscus shape in the region close to the optical axis, while the surface of each of the fifth and sixth lenses, on the object side, is convex in the region close to the optical axis, and the surface of each of the fifth and sixth lenses lens, from the side of the image surface is concave in the area close to the optical axis;

and the axial distance between the object-side surface of the first lens and the image surface is less than about 4.5 mm.

This configuration of the optical unit, in contrast to the EDoF systems of the prior art (US S8559118) disclosed in detail in this description, allows you to get a large depth of the imaged space (from 200 mm to infinity), a smooth MTF function, in contrast to the prior art, which has a pronounced maximum in the MTF function, with this example of an optical unit, the MTF function in the sagittal and meridional sections practically coincide, as clearly illustrated in Fig.20c.

FIG. 20b is a graph illustrating the MTF function of modulation gain (T) versus prior art defocusing (z).

FIG. 20c is a graph illustrating the MTF function characterizing modulation gain (T) versus defocus (z) of an inventive imaging device (EDoF).

The inventors, when developing the optical part of the imaging device, considered the possibility of making at least one of the lenses included in the optical unit having an aspherical and/or Zernike surface.

In this case, the profiles of the aspherical surfaces of the lenses of the claimed imaging device are described by the following expression (3):

where,

z - axial coordinate of a surface point relative to the surface vertex, r - radial distance from the optical axis, k - conical constant, c - surface curvature, A, B, S, D, E, F, G, H, J - aspherical coefficients.

In this case, for k=0, the surface has a spherical profile,

-1˂ to ˂0 the surface has the shape of an ellipsoid with the main axis on the optical axis (prolate ellipsoid (spheroid)

k \u003d -1 the surface has the shape of a paraboloid

k˂-1 surface has the form of a hyperboloid.

The complex shape of the lens surfaces, including aspherical ones, calculated according to expression (3) provides aberration correction and improves the quality of the image generated by the claimed device.

It should be noted that the use of aspherical surfaces provides correction of field aberrations with a small number of lenses, in this case five or six, ensuring the compactness of the patented device.

At the same time, lenses with aspherical surfaces made of optical plastic according to the invention are easily molded, which leads to a reduction in their cost.

When manufacturing lenses with aspherical surfaces from optical glass, the order of aspherical surfaces is chosen to be lower due to the higher refractive index of the glass and the low Abbe number, which also plays a role in reducing the cost of manufacturing the imaging device according to the invention and smartphones using these devices.

The surface profiles of the Zernike lenses of the claimed imaging device are described by the following expression (4):

where,

z - coordinate of a surface point along the optical axis relative to the surface vertex, r - radial distance from the optical axis, k - conical constant, c - surface curvature at the vertex, ZP _j - jth Zernike polynomial (range j from 1 to 66),

C _(j+1) - coefficient at ZP _j.

The execution of the lens surfaces described by expression (4), i.e. Zernike surfaces, provides correction of aberrations and improves the quality of the image generated by the claimed device according to the invention.

It should be noted that the use of Zernike surfaces provides correction of field aberrations with a small number of lenses, in this case five or six, ensuring the compactness of the patented device.

At the same time, the manufacture of these lenses is a rather complicated process, but the contribution to the quality of the formed image is very large.

In FIG. 21 are graphs illustrating the MTF functions characterizing modulation gain (T) versus defocus (z) for an optical assembly having an aspherical surface and/or a Zernike surface according to an embodiment of the claimed imaging device (EDoF).

As can be seen from the graphs, the use of the Zernike surface provides a more uniform modulation transfer function in the range of distances to the object from 200 mm to 5000 mm (corresponding depth of field from about -0.03 mm to 0.03 mm) compared to the aspherical surface.

On figa presents examples of plots of the modulation transfer function MTF for the claimed optical unit, characterizing the dependence of the modulation transfer coefficient (T) from defocusing (z) at a frequency of 100 lines/mm for points with coordinates of 0, 1.9 and 2.5 mm. In this case, the solid line indicates the meridional section, and the dotted line indicates the sagittal section.

On fig.22b presents the corresponding graphs of the scattering function for a point on the axis at a distance to the object (target object) d equal to 200mm, 400mm, 5000mm. In this case, on the graphs, x is the coordinate in the image plane, I is the value of the scattering function. It should be noted that for a distant object (d=5000mm), side maxima are practically absent, which provides a natural bokeh effect (without images of point objects in the form of rings and ovals).

On figs presents examples of graphs illustrating the distortion of the optical unit according to the embodiment of the claimed imaging device (EDoF).

The abscissa shows the distortion value in percent, and the ordinate shows the image size in mm. In one embodiment, the maximum image size of 2.78 mm corresponds to a 40 degree field of view (80 degree field of view). In this case, for the optical unit, the distortion does not exceed 2%.

The preferred embodiment of the inventive imaging device (see FIG. 18) is characterized by a scatter function (PSF) at an out-of-focus position with minimal side lobes, as shown in the graph of FIG. 24a. The graph clearly illustrates the maximum value P of sidelobe peaks (sidelobe level or sidelobes value) which is less than 0.1*A, where A is the maximum value of the PSF function (or central maximum), i.e. P≤0.1*A, which leads to the absence of halos in the form of rings or semirings on the image of the points of the object.

This effect is achieved through the use of a multifocal optical system with multiple pupil zones. At the same time, as a result, at the output of the optical image processing unit, a natural bokeh effect is obtained in the intermediate blurred image of the object.

The preferred embodiment of the inventive imaging device (see FIG. 18) is also characterized by a modulation transfer function (MTF).

Fig. 24b is a graph of the MTF function versus defocus (depth of field) illustrating the smooth curve of the MTF function. In this case, the unevenness U of the MTF function depending on the focal lengths in a given range of depth of field determined by the focal lengths is less than 20% in the frequency range from 0.5N to 0.9N, where N is the Nyquist frequency.

In this case, the unevenness U is determined according to the following expression (5):

U=(max-min)/(max+min)*100%,

where max is the maximum value of the MTF function in the given range,

min - the minimum value of the MTF function in the specified range.

In this case, the specified range of depths of field is from -0.03 mm to 0.03 mm, and the corresponding depth of the imaged space is from 200 mm to 5000 mm.

The graphs shown in figa, 24b, obtained on the basis of experimental data conducted by the authors during the preparation of the invention.

Thus, the inventors have developed a method and apparatus for imaging with increased depth of imaging space, in which:

- no need to move optical elements,

- no complex optics required (additional optical elements),

-providing an optical device with an increased depth of image space without autofocus function,

-no focus delay

-fast recovery algorithm,

- no need for user interaction,

- no control over the process of taking an image by a smartphone user is required (i.e. the user does not need to select the focus area),

- the object, regardless of the distance to the camera, is always in focus,

- a simple image data processing algorithm,

- inexpensive, compact imaging device, with the possibility of application for mass production.

Fig. 23a schematically shows an optical unit with a division of the entrance pupil into two pupil zones (PZ1, PZ2) and a receiver according to an embodiment of an imaging device. Beams of rays enter the optical unit, conventionally represented as a single lens, through the entrance pupil, located in the space of objects (objects), and divided into two zones. Two parallel beams of rays (the number of beams of rays corresponds to the number of pupillary zones and their number is not limited to the example shown in figa) pass through the optical unit in the form of a set of lenses (one lens is conventionally shown in figa). In this case, the point of intersection of the beams in the beam sets the focal length EFL (EFL1 for the first beam of beams L1 and EFL2 for the second beam of beams L2), measured from the main plane of the optical unit to the point of intersection of the beams in the beam along the optical axis. Changing the optical power P or the focal length EFL depending on the coordinates of the pupil (x, y) sets the optical power distribution map or the focal length distribution map, respectively, presented in Fig.23b and 23d. The receiver is located at a distance z' from the optical unit and is configured to capture images generated by the optical unit through the pupillary areas (PZ1,PZ2). In this case, different focal lengths correspond to different distances z in the space of objects, where z=z'/EFL ² . Thus, the receiver has information about targets at various distances, but the image formed at the receiver is blurry, as has already been detailed in the description. In this case, the level of "blurring" of the image is characterized by the scattering function (PSF).

Power distribution maps of the optical unit according to Fig. 23a of the imaging device according to the invention are shown in Figs. 23b, 23c, 23d. A power distribution map (P) can be represented as a function of pupil coordinates versus focal length (EFL) or P=1/EFL, as can be clearly seen in the graphs of Fig. 23b, Fig. 23c. In FIG. 23d is a map of focal length distribution as a function of pupil coordinates. On figs presents the distribution of the focal length in the sagittal section of the pupil (ie, this is the distribution profile of the focal length on the map of Fig. 23d with y coordinate on the pupil equal to zero).

The blur level can be described by the scattering function PSF (x, y), where x, y are coordinates at the receiver. The scattering function depends on the distance to the object (object) and the angle of the field of view at which the object is observed. The function PSF (x, y) is determined by the shape of the pupil and aberration properties (for example, wave aberrations (OPD)). Wave aberrations (OPD) of each pupil zone determine or set the image quality in accordance with the characteristics of the optical system for the distance range d1, d2, d3…dN. The PSF describes the response of an optical block to a point object. The derivation of the scattering function ( PSF ) is detailed in the book: Goodman JW "Introduction to Fourier Optics" second edition, New York: McGraw-Hill, 1996, p.145

(или интенсивности объекта, приведенной к приемнику, которая соответствует идеальному изображению объекта) и функции

PSF:. Следует отметить, что λ в нижнем индексе означает зависимость от длины волны. Учитывая чувствительность приемника к длине волны

т.е. спектральную чувствительность, сформированное изображение Intensity distribution or intensity of the image Ic as a function of coordinates in the image (hereinafter referred to as the image Ic) formed by the optical unit on at least a part of the receiver, within which the scattering function can be considered invariant (i.e. the scattering function does not depend on the field angle vision) can be represented as a convolution of the intensity distribution of the object

(or the intensity of the object, reduced to the receiver, which corresponds to the ideal image of the object) and the function

PS:. It should be noted that λ in the subscript denotes wavelength dependence. Given the sensitivity of the receiver to the wavelength

those. spectral sensitivity, formed image

- интенсивность объекта в зависимости от длины волны (λ), сформированного на по меньшей мере части приемника,

- функция рассеяния в зависимости от длины волны (λ),

- чувствительность приемника к длине волны (λ), d_λ - дифференциал длины волны излучения. where Ic is the intensity of the image (hereinafter referred to as the image) formed on at least a part of the receiver,

- the intensity of the object depending on the wavelength (λ) formed on at least part of the receiver,

- scattering function depending on the wavelength (λ),

- sensitivity of the receiver to the wavelength (λ), d _λ - radiation wavelength differential.

где S - оператор оцифровки, η - шум, Ic - интенсивность изображения, сформированного на по меньшей мере части приемника. Полученное изображение отправляется в блок обработки изображения.The image Ic formed at the receiver is digitized, i.e. becomes discrete with the pixel repetition rate of the receiver, is level-quantized (preferably 256 quantization levels), integrated pixel-by-pixel at the receiver, and becomes noisy η, which gives the resulting (intermediate) image:

where S is the digitization operator, η is the noise, Ic is the intensity of the image formed on at least part of the receiver. The resulting image is sent to the image processing unit.

и привносит (добавляет) шум (η) в промежуточное изображение (фиг.25d). Изображение, сформированное на приемнике, оцифровывается (S). На фиг.25e представлено получение результирующего промежуточного изображения на приемнике On figa-e schematically shows the process of imaging, occurring in the optical unit and the receiver. In Fig.25a, the object under consideration is conditionally divided into parts, within which the scattering function can be considered invariant, i.e. the PSF function is considered as an invariant. Mathematically, the optical unit of the imaging device can be represented as a convolution of the intensity distribution on the object and the scattering function (Fig.25b). The PSF function is calculated for each part of the receiver (see Fig.25c). The receiver is characterized by spectral sensitivity

and brings (adds) noise (η) to the intermediate image (Fig.25d). The image formed at the receiver is digitized (S). Figure 25e shows the acquisition of the resulting intermediate image at the receiver

и идеальным изображением

объекта минимизируется, например :

.FIG. 26 is a schematic view of an object image (Yc) obtained at the receiver (see the resulting intermediate image in FIG. 25d and the reconstructed image obtained after processing in the image processing unit. Thus, using the information of the registered image (Yc) by the receiver and dependence of the scattering function on the coordinates at the PSF receiver (x, y), the image processing unit is configured to process the image in such a way that the discrepancy or discrepancy between the reconstructed image

and perfect picture

object is minimized, for example:

.

Industrial Applicability

The device and method for imaging with increased image depth can be used in compact, portable devices, such as tablet computers, laptops, teleconferencing systems, in addition, it can be used for photo and video recording by EDoF cameras of smartphones running Android, iOS operating systems, HarmonyOS, as well as systems similar to them, and in other devices in which there is a need to provide real-time shooting of objects at different distances. In addition, it can be used in security devices, video surveillance cameras, devices for taking biometric data, in microscopy, in security systems for scanning the retina of moving objects with instant fixation of an object without focusing it and non-contact fingerprinting.

Claims (79)

1. An image forming device with an increased depth of the imaged space, comprising: - an optical unit configured to simultaneously form intermediate images of an object at different distances with the effect of blurring at least parts of the object images, and containing at least one optical element, and having at least two pupil zones formed in such a way as to provide a preliminary a given distribution of optical powers and aberrations within each of at least two pupil zones, on the basis of which a scattering function is formed, characterized by a curve with minimized side maxima compared to the central maximum in a given range of distances to the object, with each of at least two zones of the pupil correspond to the given corresponding range of distances to the object and the given corresponding range of angles of the field of view, - a receiver configured to register intermediate images formed by the optical unit from different distances to the object and at different angles of the field of view simultaneously, - an image processing unit connected to the receiver and configured to process intermediate images of an object with a blur effect of at least parts of the object images recorded by the receiver, while processing these images is performed based on the obtained scattering function in a given range of distances and angles of the field of view, with restoring the output of the resulting images without the blur effect, regardless of the distances to the object. 2. The device according to claim 1, in which the minimized lateral maxima of the scattering function formed by the optical unit do not exceed 10% of the central maximum in a given range of distances to the object. 3. The device according to claim. 1 or 2, in which the specified range of distances to the object is from 400 to 5000 mm. 4. The device according to claim. 2, in which the specified range of angles of the field of view is from -40 to +40 degrees. 5. The device according to one of paragraphs. 1-4, in which the optical block is designed in such a way that provides the formation of an invariant scattering function that does not depend on the distance to the object in a given range. 6. The device according to one of paragraphs. 1-4, in which the optical unit is designed in such a way that provides the formation of an invariant scattering function that does not depend on the angle of the field of view. 7. The device according to one of paragraphs. 1-4, in which the optical unit is designed in such a way that provides the formation of an invariant scattering function that does not depend on the angle of the field of view and the distance to the object. 8. The device according to one of paragraphs. 1-4, in which the optical unit is configured to generate a uniform modulation transfer function (MTF) independent of the distance to the object. 9. The device according to one of the previous claims, in which the optical unit is additionally configured to simultaneously form intermediate images of the object at different distances with the effect of blurring the entire image of the object. 10. The device according to claim 1, in which the optical unit contains at least one optical element, which is at least one of: lenses with different surface profiles, a composite lens, a diffractive optical element, a holographic optical element, a polarizing element, an amplitude- phase mask. 11. The device according to claim. 10, in which at least one lens has an aspherical surface and/or, in particular, a Zernike surface. 12. An apparatus according to claim 11, wherein lenses with an aspherical surface and/or in particular a Zernike surface provide aberration correction. 13. The device according to claim 10, in which the optical elements of the optical unit are located along the optical axis from the side of the object to the image surface: the first lens, the second lens, the third lens, the fourth lens, the fifth lens and the sixth lens, while the first lens has a positive power, wherein the surface of the first lens on the object side is convex and the surface of the first lens on the side of the image surface is concave; the second lens has a negative optical power, while the surface of the second lens on the object side is concave and the surface of the second lens on the side of the image surface is also concave; the third lens has a positive optical power, the fourth lens has the shape of a meniscus, the fifth lens and the sixth lens each have a meniscus shape in the area close to the optical axis, while the surface of each of the fifth and sixth lenses on the object side is convex in the area close to the optical axis, and the surface of each of the fifth and sixth lenses with side of the image surface is concave in the region close to the optical axis. 14. The device of claim 10, wherein the composite lens is at least one of: a bifocal lens, a progressive lens, a Fresnel lens, and/or combinations thereof. 15. The device of claim. 10, in which the diffractive optical element is a diffractive microstructure, configured to provide amplitude-phase modulation of transmitted or reflected radiation. 16. The apparatus of claim 10, wherein the holographic element is designed and positioned to contribute to the elimination of chromatic aberrations. 17. The apparatus of claim 10 wherein the polarizing element is a polarizer and/or a phase plate and/or a polarizing filter. 18. The device according to claim 17, in which the polarization element, which is a polarizer and/or a phase plate, is configured to provide multiple passages of radiation through the optical elements of the optical unit. 19. The apparatus of claim 10 wherein the polarizing element is a polarizing filter capable of eliminating glare. 20. The device according to claim 10, in which the amplitude-phase mask is made and placed in the optical block in such a way as to provide a phase delay or amplitude modulation of the radiation to ensure a uniform modulation gain over the required image depth. 21. The device according to claim 10, in which the amplitude-phase mask is one of: a mask with a phase profile of a fourth degree parabola, a cubic phase mask, a mask with concentric annular holes in the screen. 22. The device according to claim. 1, in which each of the optical elements of the optical block is made of an optically transparent material selected from at least one of: optical glass, optical crystal and polymer. 23. The apparatus of claim. 1, further comprising a shutter located in the plane of the aperture diaphragm or its conjugate plane of the imaging device. 24. The device according to claim 23, in which the shutter is configured to regulate the opening or closing of a given number of pupil zones. 25. The apparatus of claim. 1, wherein the receiver is a matrix photodetector. 26. The device according to claim 25, in which the matrix photodetector is configured to detect electromagnetic radiation in the range of 0.4-0.7 microns, and the sensor pixel size is from 0.7 to 1.5 microns. 27. The apparatus of claim 25, wherein the receiver is a CMOS sensor or CCD receiver. 28. The device according to claim 1, in which the image processing unit is configured to process intermediate images of the object with restoring the resulting images at the output based on a convolutional neural network and / or Wiener filter, the parameters of which are preliminarily determined based on the design parameters of the optical unit and the receiver. 29. The device according to claim 1, in which the parameters of the optical block include at least one of: the radii of the surfaces of the optical elements, the coefficients of the aspherical surfaces of the optical elements, the thicknesses of the optical elements, the refractive indices of the materials of the optical elements, the dispersion value of the optical elements, the distances between the optical elements elements in the optical block, the length of the device, representing the axial distance between the surface of the first optical element on the object side and the image surface on the receiver. 30. The device according to claim 1, in which the receiver parameters include at least one of: spectral sensitivity, noise, digitization parameters. 31. An image forming device with an increased depth of the imaged space, comprising: - an optical unit configured to simultaneously form intermediate images of an object at different distances with the effect of blurring at least parts of the object images, and containing a set of optical elements, and having at least two pupil zones formed in such a way as to provide a predetermined distribution of optical forces and aberrations within each of at least two pupil zones, on the basis of which a scattering function is formed, characterized by a curve with minimized side maxima compared to the central maximum in a given range of distances to the object, while each of at least two pupil zones corresponds to a given corresponding range of distances to the object and a given corresponding range of angles of the field of view, while the set of optical elements contains located along the optical axis from the side of the object to the image surface: the first lens, the second lens, the third lens, fourth lens, fifth lens, and sixth lens, while the first lens has a positive power, wherein the surface of the first lens on the object side is convex and the surface of the first lens on the side of the image surface is concave; the second lens has a negative optical power, while the surface of the second lens on the object side is concave and the surface of the second lens on the side of the image surface is also concave; the third lens has a positive optical power, the fourth lens has the shape of a meniscus, the fifth lens and the sixth lens each have a meniscus shape in the area close to the optical axis, while the surface of each of the fifth and sixth lenses on the object side is convex in the area close to the optical axis, and the surface of each of the fifth and sixth lenses with side of the image surface is concave in the region close to the optical axis, - a receiver configured to register intermediate images formed by the optical unit from different distances to the object and at different angles of the field of view simultaneously, - an image processing unit connected to the receiver and configured to process intermediate images of an object with a blur effect of at least parts of the object images recorded by the receiver, while processing these images is performed based on the obtained scattering function in a given range of distances and angles of the field of view, with restoring the output of the resulting images without the blur effect, regardless of the distances to the object. 32. The device according to claim 31, in which the minimized side maxima of the scattering function generated by the optical unit do not exceed 10% of the central maximum in a given range of distances to the object. 33. The device according to claim 31 or 32, in which the specified range of distances to the object is from 400 to 5000 mm. 34. The apparatus of claim 32, wherein the predetermined range of field of view angles is -40 to +40 degrees. 35. The device according to one of paragraphs. 31-34, in which the optical unit is designed in such a way that provides the formation of an invariant scattering function that does not depend on the distance to the object in a given range. 36. The device according to one of paragraphs. 31-34, in which the optical block is designed in such a way that provides the formation of an invariant scattering function that does not depend on the angle of the field of view. 37. The device according to one of paragraphs. 31-34, in which the optical block is designed in such a way that provides the formation of an invariant scattering function that does not depend on the angle of the field of view and the distance to the object. 38. The device according to one of paragraphs. 31-34, wherein the optical unit is configured to form a uniform modulation transfer function (MTF) curve independent of object distance. 39. The device according to one of the previous claims, in which the optical unit is additionally configured to simultaneously form intermediate images of the object at different distances with the effect of blurring the entire image of the object. 40. The device according to claim 31, wherein each of the optical elements of the optical unit is made of an optically transparent material selected from at least one of: optical glass, optical crystal and polymer. 41. The apparatus of claim. 31, further comprising a shutter located in the plane of the aperture diaphragm or its conjugate plane of the imaging device. 42. The device of claim 41, wherein the shutter is configured to control the opening or closing of a predetermined number of pupil zones. 43. The device according to claim 31, in which the receiver is a matrix photodetector. 44. The device according to claim 43, in which the matrix photodetector is configured to detect electromagnetic radiation in the range of 0.4-0.7 microns, and the sensor pixel size is from 0.7 to 1.5 microns. 45. The apparatus of claim 31, wherein the receiver is a CMOS sensor or CCD receiver. 46. The device according to claim 31, in which the image processing unit is configured to process intermediate images of the object with restoring the output of the resulting images based on a convolutional neural network and / or a Wiener filter, the parameters of which are predetermined based on the parameters of the optical unit and the receiver. 47. The device according to claim 31, in which the parameters of the optical block include at least one of: the radii of the surfaces of the optical elements, the coefficients of the aspherical surfaces of the optical elements, the thicknesses of the optical elements, the refractive indices of the materials of the optical elements, the dispersion value of the optical elements, the distances between the optical elements elements in the optical block, the length of the device, representing the axial distance between the surface of the first optical element on the object side and the image surface on the receiver. 48. The device according to claim 31, in which the receiver parameters include at least one of: spectral sensitivity, noise, digitization parameters. 49. A method for forming an image with an increased depth of the imaged space, comprising steps in which: - provide simultaneous formation of intermediate images of the object at different distances with the effect of blurring at least parts of the images of the object by means of an optical unit having at least two pupil zones formed in such a way as to provide a predetermined distribution of optical powers and aberrations within each of the at least two pupil zones, on the basis of which a scattering function is formed, characterized by a curve with minimized lateral maxima compared to the central maximum in a given range of distances to the object, while each of at least two pupil zones corresponds to a given corresponding range of distances to the object and a given appropriate range of field of view angles, - provide registration of intermediate images formed by the optical unit, from different distances to the object and at different angles of the field of view simultaneously, by means of a receiver, - the image processing unit processes the intermediate images of the object with the effect of blurring at least parts of the images of the object, based on the obtained scattering function in a given range of distances and angles of the field of view, and - restore the resulting images without the blur effect, regardless of the distances to the object. 50. The method according to claim 49, wherein the minimized side maxima of the scattering function generated by the optical unit do not exceed 10% of the central maximum in a given range of distances to the object. 51. The method according to claim 49 or 50, in which the specified range of distances to the object is from 400 to 5000 mm. 52. The method of claim 49, wherein the predetermined range of field of view angles is -40 to +40 degrees. 53. The method according to one of paragraphs. 49-52, in which the optical unit is designed in such a way that provides the formation of an invariant scattering function that does not depend on the distance to the object in a given range. 54. The method according to one of paragraphs. 49-52, in which the optical unit is designed in such a way that provides the formation of an invariant scattering function that does not depend on the angle of the field of view. 55. The method according to one of paragraphs. 49-52, in which the optical unit is configured to form a uniform modulation transfer function (MTF) independent of the distance to the object. 56. The method according to one of the preceding claims, in which the optical unit is additionally configured to simultaneously form intermediate images of the object at different distances with the effect of blurring the entire image of the object. 57. The method according to claim 49, in which the processing of intermediate images of the object is performed with the restoration of the resulting images at the output by the image processing unit based on a convolutional neural network and / or Wiener filter, the parameters of which are preliminarily determined based on the parameters of the optical unit and the receiver. 58. The method according to claim 57, in which the parameters of the optical block include at least one of: the radii of the surfaces of the optical elements, the coefficients of the aspherical surfaces of the optical elements, the thicknesses of the optical elements, the refractive indices of the materials of the optical elements, the dispersion value of the optical elements, the distances between the optical elements elements in the optical block, the length of the device, representing the axial distance between the surface of the first optical element on the object side and the image surface on the receiver. 59. The method of claim 57, wherein the receiver parameters include at least one of: spectral sensitivity, noise, sampling parameters.

Images