WO2023029520A1

WO2023029520A1 - Method and apparatus for light-field-scanning imaging system to photograph dynamic scene

Info

Publication number: WO2023029520A1
Application number: PCT/CN2022/088564
Authority: WO
Inventors: 乔晖; 戴琼海; 吴嘉敏
Original assignee: 清华大学
Priority date: 2021-08-31
Filing date: 2022-04-22
Publication date: 2023-03-09
Also published as: CN113487658A; CN113487658B

Abstract

Disclosed in the present application are a method and apparatus for a light-field-scanning imaging system to photograph a dynamic scene. The method comprises: collecting an original light field image by means of a light field scanning system; transforming the original light field image to obtain a three-dimensional image stack; separately processing images in the three-dimensional image stack, and combining same to obtain image pairs; performing registration on the image pairs, and performing calculation to obtain a coordinate transformation relationship between the image pairs; and performing scatter interpolation according to the coordinate transformation relationship, so as to obtain a high-resolution light-field-scanned single-visual-angle image. By means of the method in the present application, artifact-free high-resolution photography for a dynamic scene can be performed by means of light field scanning.

Description

A dynamic scene shooting method and device for scanning light field imaging system

Cross References to Related Applications

This application claims the priority of the Chinese patent application number "202111014061.X" submitted by Tsinghua University on August 31, 2021, the application name is "A method and device for shooting dynamic scenes for scanning light field imaging system" .

technical field

The present application relates to the technical field of scanning light field imaging systems, and in particular to a dynamic scene shooting method for scanning light field imaging systems.

Background technique

In recent years, ultra-high pixel count imaging has gradually entered people's field of vision. With the popularization and proliferation of industries or equipment such as machine vision, drones, and high-definition surveillance systems, people's demand for high-resolution, large-pixel imaging has become increasingly apparent.

The three limiting conditions that determine the imaging quality or imaging resolution of the imaging system are as follows: First, the sampling rate of the image sensor: the popular image sensors are mainly divided into two types: CCD and CMOS. Being smaller is conducive to the generation of higher-resolution images; the second is the optical diffraction limit of the system: according to the Rayleigh resolution criterion, the numerical aperture of each lens in the imaging system determines the resolution of the system; the third is aberration, image Differences generally exist in the natural environment, such as atmospheric scattering. At the same time, aberrations also exist in the lens. Since the lens is a manually polished device, there will be deviations from the ideal lens model in theoretical optics. On the other hand, as the size of the lens gradually increases, the paraxial optical theory in the ideal optical system is no longer applicable, and the trajectory of off-axis rays is difficult to predict as easily as that of paraxial rays. The above three points all limit the imaging capability of the system, further hindering the development of gigapixel imaging.

Thanks to the development of the industry, the diffraction limit that the optical system can achieve is enough to meet the needs of our macro scene shooting, and there is also enough technology to produce a high-resolution image sensor with a large area array and a small pixel size. What hinders the further development of these processes is the existence of aberrations. In the physical model, as the size of a single lens increases, the aberration will gradually increase. Therefore, as the number of pixels increases, the number of effective pixels will be limited to a limited scale. Under such conditions, no matter how the number of pixels of the image sensor or the numerical aperture of the lens is increased, it is difficult to further improve the resolution and clarity of the image.

Under such conditions, the development of Gigapixel has gone through two stages. In the first stage, by reducing the size of the optical aperture and reducing the existence of aberrations, it can break through the limitation of the number of effective pixels of the original ordinary camera. However, this solution will lead to a decrease in the amount of light passing through, an increase in the exposure time, and a significant decrease in the signal-to-noise ratio. In the second stage, multi-lens shooting is stitched together to form a billion-pixel imaging. By increasing the number of lenses, this solution reduces the corresponding increase in aberration with the increase in the size of a single lens. Compared with the method of reducing the aperture, it breaks through the limitation of aberrations and achieves better imaging effects. However, neither of these two solutions actually solves the problem of aberration, and the best imaging effect that can be achieved is far less than the performance of the optical diffraction limit.

The scanning light field system can break the mutual constraints between spatial resolution and angular resolution by using microlenses for dense spatial moving sampling. Obtain multi-angle sampling while achieving high spatial resolution. However, the scanning light field imaging system sacrifices a certain time resolution to achieve high-resolution imaging through the fusion between multiple frames.

The scanning light field imaging technique mentioned above obtains high spatial resolution by sacrificing temporal resolution. Therefore, serious artifacts will appear when shooting high-speed dynamic scenes, making it impossible to perform high-resolution imaging with multi-frame fusion of scanning light fields.

Contents of the invention

This application aims to solve one of the technical problems in the related art at least to a certain extent.

Therefore, the first purpose of the present application is to propose a dynamic scene shooting method for a scanning light field imaging system, which can use the scanning light field to perform artifact-free high-resolution shooting of dynamic scenes.

The second purpose of the present application is to propose a device for reconstructing geometric motion details by fitting a depth image.

To achieve the above purpose, the embodiment of the first aspect of the present application proposes a dynamic scene shooting method for a scanning light field imaging system, including:

S1, the original image of the light field is acquired by scanning the light field system;

S2, transforming the original image of the light field to obtain a three-dimensional image stack;

S3, separately processing the images in the three-dimensional image stack, and combining them to obtain an image pair;

S4, registering the image pair, and calculating a coordinate transformation relationship of the image pair;

S5. Perform scatter interpolation according to the coordinate transformation relationship to obtain a high-resolution scanning light field single-view image.

In addition, the dynamic scene shooting method for the scanning light field imaging system according to the above-mentioned embodiments of the present application may also have the following additional technical features:

Further, in one embodiment of the present application, the sent method also includes:

The images of each viewing angle in the three-dimensional image stack are registered to realize dynamic scene shooting in the scanning light field.

Further, in one embodiment of the present application, the processing of the images in the three-dimensional image stack separately and combining to obtain an image pair includes:

The fused single-view images are decoupled into low-resolution single-view time series images according to the shooting time distribution, and one sequence of images is acquired and combined with images in other time series.

Further, in one embodiment of the present application, the registering the image pair includes:

A certain time frame in a scanning period is used as a standard, and the registration estimation is performed on the standard with other time frames in the scanning period.

Further, in one embodiment of the present application, the scatter interpolation is performed according to the coordinate transformation relationship to obtain a high-resolution scanning light field single-view image, including:

Scatter interpolation is performed on the fused single-view image and the coordinate transformation relationship to obtain the normally distributed high-resolution scanning light field single-view image.

In the dynamic scene shooting method used in the scanning light field imaging system of the embodiment of the present application, the original image of the light field is obtained by scanning the light field system, the original image of the light field is transformed to obtain a three-dimensional image stack, and the images in the three-dimensional image stack are processed. Separately process and combine to obtain image pairs, register the image pairs, calculate the coordinate transformation relationship of the image pairs, and perform scatter interpolation according to the coordinate transformation relationship to obtain high-resolution scanning light field single-view images. By adopting the above method, the present application can use the scanning light field to perform artifact-free high-resolution shooting of dynamic scenes.

To achieve the above purpose, the embodiment of the second aspect of the present application proposes a dynamic scene shooting device for a scanning light field imaging system, including:

The acquisition module is used to acquire the original image of the light field through the scanning light field system;

A transformation module, configured to obtain a three-dimensional image stack by transforming the original image of the light field;

A combining module, configured to process the images in the three-dimensional image stack separately and combine them to obtain an image pair;

A calculation module, configured to register the image pair, and calculate the coordinate transformation relationship of the image pair;

The single-view module is used to perform scatter interpolation according to the coordinate transformation relationship to obtain a high-resolution scanning light field single-view image.

The dynamic scene shooting device used in the scanning light field imaging system of the embodiment of the present application obtains the original image of the light field through the scanning light field system, converts the original image of the light field to obtain a three-dimensional image stack, and performs the image processing in the three-dimensional image stack Separately process and combine to obtain image pairs, register the image pairs, calculate the coordinate transformation relationship of the image pairs, and perform scatter interpolation according to the coordinate transformation relationship to obtain high-resolution scanning light field single-view images. By adopting the above method, the present application can use the scanning light field to perform artifact-free high-resolution shooting of dynamic scenes.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a flowchart of a dynamic scene shooting method for a scanning light field imaging system according to an embodiment of the present application.

Fig. 2 is a schematic structural diagram of a dynamic scene shooting device used in a scanning light field imaging system according to an embodiment of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, and are intended to explain the present application, and should not be construed as limiting the present application.

The dynamic scene shooting method and device for the scanning light field imaging system according to the embodiments of the present application will be described below with reference to the accompanying drawings.

The dynamic scene shooting method used in the scanning light field imaging system of the embodiment of the present application solves the problem that the traditional scanning light field imaging technology obtains high spatial resolution by sacrificing time resolution, so serious artifacts will appear when shooting high-speed dynamic scenes , leading to the technical problem that high-resolution imaging of multi-frame fusion of scanning light field cannot be performed. This application can use scanning light field to perform artifact-free high-resolution shooting of dynamic scenes.

FIG. 1 is a flow chart of a dynamic scene shooting method for a scanning light field imaging system provided by an embodiment of the present application.

As shown in Figure 1, the dynamic scene shooting method for the scanning light field imaging system includes:

In step S1, the original image of the light field is acquired by scanning the light field system.

Specifically, light field original images containing different angle information are collected through a light field system or a scanning light field system.

Step S2, transforming the original image of the light field to obtain a three-dimensional image stack.

It can be understood that the light field original image can be transformed into a three-dimensional image stack with different angle information by rearranging or other methods.

In step S3, the images in the three-dimensional image stack are processed separately and combined to obtain an image pair.

It can be understood that the images of different viewing angles in step S2 are processed separately. The processing method is as follows: decouple the fused single-view images into low-resolution single-view time series images according to the shooting time distribution, take one of the images as an example, and combine them with images in other time series in pairs.

Step S4, registering the image pair, and calculating the coordinate transformation relationship of the image pair.

It can be understood that, by taking a certain time frame in a scan cycle as a standard, and performing registration estimation with other time frames in the scan cycle (for example, there are 24 other frames in a 5*5 cycle scan), Calculate the coordinate transformation relationship between two images.

Step S5, performing scatter interpolation according to the coordinate transformation relationship to obtain a high-resolution scanning light field single-view image.

It can be understood that, after the image pairs in step S3 are registered or optical flow estimated, and the coordinate transformation relationship of the scene in the image pair is calculated, the fused single-view image and the calculated coordinate transformation are performed by the scatter interpolation method Scatter point interpolation is carried out according to the relation, and a high-resolution scanning light field single-view image with normal distribution is obtained.

That is to say, scatter interpolation is performed on the rearranged high-resolution artifact single-view images in one scanning period according to the coordinate transformation relationship, and the optimal artifact-free high-resolution single-view images are estimated. It should be noted that the relationship between images from different perspectives is decoupled from each other. Separate operations and processing are required for different perspectives.

The embodiment of the present application also includes: registering the images of each perspective in the three-dimensional image stack to realize dynamic scene shooting in the scanning light field.

It should be understood that step S4 and step S5 are performed for each viewing angle, and by analogy, scanning light field dynamic scene shooting can be realized.

As shown in FIG. 2, the dynamic scene shooting device 10 for the scanning light field imaging system includes:

The acquisition module 100 , the transformation module 200 , the combination module 300 , the calculation module 400 and the single view module 500 .

The acquisition module 100 is used to acquire the original image of the light field by scanning the light field system;

The addition module 200 is used for transforming the original image of the light field to obtain a three-dimensional image stack;

A combination module 300, configured to process the images in the three-dimensional image stack separately, and combine them to obtain an image pair;

Calculation module 400, for registering the image pair, and calculating the coordinate transformation relationship of the image pair;

The single-view module 500 is configured to perform scatter interpolation according to the coordinate transformation relationship to obtain a single-view image of a high-resolution scanning light field.

Further, the device 10 also includes:

The dynamic scene module is used for registering the images of each perspective in the three-dimensional image stack to realize dynamic scene shooting in the scanning light field.

Further, the above combination module 300 is specifically used for:

Further, the device 10 also includes:

The registration module is configured to use a certain time frame in a scanning cycle as a standard, and perform registration estimation on the standard and other time frames in the scanning cycle.

Further, the above-mentioned single view module 500 is specifically used for:

Scatter interpolation is performed on the fused single-view image and the coordinate transformation relationship to obtain a normal-distributed high-resolution scanning light field single-view image.

According to the dynamic scene shooting device for the scanning light field imaging system proposed in the embodiment of the present application, the original image of the light field is obtained by scanning the light field system, and the original image of the light field is transformed to obtain a three-dimensional image stack, and the three-dimensional image stack is obtained The images are processed separately, combined to obtain image pairs, the image pairs are registered, and the coordinate transformation relationship of the image pairs is calculated, and the scatter interpolation is performed according to the coordinate transformation relationship to obtain a high-resolution scanning light field single-view image. By adopting the above method, the present application can use the scanning light field to perform artifact-free high-resolution shooting of dynamic scenes.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims

A dynamic scene shooting method for a scanning light field imaging system, characterized in that the method comprises the following steps:

The original image of the light field is acquired by scanning the light field system;

Transforming the original image of the light field to obtain a three-dimensional image stack;

processing the images in the three-dimensional image stack separately, and combining them to obtain an image pair;

Registering the image pair, and calculating the coordinate transformation relationship of the image pair;

Scatter interpolation is performed according to the coordinate transformation relationship to obtain a high-resolution scanning light field single-view image.
The dynamic scene shooting method for a scanning light field imaging system according to claim 1, wherein the method further comprises:

The images of each viewing angle in the three-dimensional image stack are registered to realize dynamic scene shooting in the scanning light field.
The dynamic scene shooting method for a scanning light field imaging system according to claim 1, wherein said processing the images in the three-dimensional image stack separately and combining them to obtain an image pair comprises:

The fused single-view images are decoupled into low-resolution single-view time series images according to the shooting time distribution, and one sequence of images is obtained and the images in other time series are combined in pairs.
The dynamic scene shooting method for a scanning light field imaging system according to claim 1, wherein said registering said pair of images comprises:

A certain time frame in a scanning period is used as a standard, and the registration estimation is performed on the standard with other time frames in the scanning period.
The dynamic scene shooting method for a scanning light field imaging system according to claim 1, wherein the step of performing scatter interpolation according to the coordinate transformation relationship to obtain a high-resolution scanning light field single-view image comprises:

Scatter interpolation is performed on the fused single-view image and the coordinate transformation relationship to obtain the normally distributed high-resolution scanning light field single-view image.
A dynamic scene shooting device for a scanning light field imaging system, characterized in that it includes:

The acquisition module is used to acquire the original image of the light field through the scanning light field system;

A transformation module, configured to obtain a three-dimensional image stack by transforming the original image of the light field;

A combining module, configured to process the images in the three-dimensional image stack separately and combine them to obtain an image pair;

A calculation module, configured to register the image pair, and calculate the coordinate transformation relationship of the image pair;

The single-view module is used to perform scatter interpolation according to the coordinate transformation relationship to obtain a high-resolution scanning light field single-view image.
The dynamic scene shooting device for a scanning light field imaging system according to claim 6, further comprising:

The dynamic scene module is configured to register each view image in the three-dimensional image stack to realize dynamic scene shooting in the scanning light field.
The dynamic scene shooting device for a scanning light field imaging system according to claim 6, wherein the combination module is used for:

The fused single-view images are decoupled into low-resolution single-view time series images according to the shooting time distribution, and one sequence of images is obtained and the images in other time series are combined in pairs.
The dynamic scene shooting device for scanning light field imaging system according to claim 6, further comprising:

The registration module is configured to use a certain time frame in a scanning cycle as a standard, and perform registration estimation on the standard with other time frames in the scanning cycle.
The dynamic scene shooting device for a scanning light field imaging system according to claim 6, wherein the single-view module is used for:

Scatter interpolation is performed on the fused single-view image and the coordinate transformation relationship to obtain the normally distributed high-resolution scanning light field single-view image.