CN116819879A

CN116819879A - Light field acquisition structure, method and equipment

Info

Publication number: CN116819879A
Application number: CN202310796591.7A
Authority: CN
Inventors: 李亚鹏; 董学; 张�浩; 李扬冰; 胡飞涛
Original assignee: BOE Technology Group Co Ltd; Beijing BOE Technology Development Co Ltd
Current assignee: BOE Technology Group Co Ltd; Beijing BOE Technology Development Co Ltd
Priority date: 2023-06-30
Filing date: 2023-06-30
Publication date: 2023-09-29

Abstract

The embodiment of the application provides a light field acquisition structure, a method and equipment, wherein the light field acquisition structure comprises the following components: a gating window layer, a main lens, an imaging unit array, and an imaging sensor; the main lens, the imaging unit array and the imaging sensor are sequentially arranged at intervals; the gating window layer is provided with a gating window, and the gating window layer is configured to: restricting a transmission region of light on the main lens through the gating window; the gating window is a light-transmitting area on the gating window layer; the image sensor is configured to sense an optical signal incident to the transmission region and transmitted through the main lens, the imaging unit array in sequence. The light field acquisition structure provided by the embodiment of the application can solve the problem of lower resolution of light field images.

Description

Light field acquisition structure, method and equipment

Technical Field

The present application relates to the field of optical imaging technology, and in particular, to a light field acquisition structure, method and apparatus.

Background

The light field can acquire three-dimensional information with higher precision than the traditional two-dimensional imaging and the traditional three-dimensional imaging represented by binocular stereoscopic vision by recording light ray data with higher dimensionality, so that a dynamic environment can be accurately perceived.

Compared with the traditional video, the light field image can provide more realistic viewing experience, and meanwhile, the light field is also a key for solving convergence focusing conflict in a VR (Virtual Reality) scene.

The conventional integrated light field camera is represented by lytro (a light field camera), and the lytro camera records the light angle of each view point by adding a micro lens array on the focal plane of a main lens, so that the view point images can be recovered, and the functions of refocusing, depth estimation and the like are realized.

However, since the lytro camera sacrifices the resolution of the image in order to record information of multiple viewpoints, the resolution of the single viewpoint image is low, and the user experience is greatly affected.

Disclosure of Invention

The embodiment of the application aims to provide a light field acquisition structure, a light field acquisition method and light field acquisition equipment, which are used for solving the problem of lower resolution of light field images. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present application provides a light field acquisition structure, including: a gating window layer, a main lens, an imaging unit array, and an imaging sensor; the main lens, the imaging unit array and the imaging sensor are sequentially arranged at intervals;

the gating window layer is provided with a gating window, and the gating window layer is configured to: restricting a transmission area of light on the main lens through the gating window; the gating window is a light-transmitting area on the gating window layer;

the image sensor is configured to sense an optical signal incident to the transmission region and transmitted through the main lens, the imaging unit array in sequence.

Optionally, the light field acquisition structure further comprises: a gating window control unit;

the gating window control unit is configured to apply a first control signal to the gating window layer, the first control signal being used to control the shape, position and/or aperture size of the gating window.

Optionally, the imaging unit array is a liquid crystal lens array;

the light field acquisition structure further comprises: a lens control unit;

the lens control unit is configured to apply a second control signal to the liquid crystal lens array, the second control signal being used to control an aperture size of a liquid crystal lens in the liquid crystal lens array.

Optionally, an aperture size A of the gating window and a focal length F of the main lens ₁ An aperture size l of an imaging unit in the imaging unit array, a focal length F of the imaging unit ₂ The relationship between them satisfies: l/F ₂ And A/F ₁ The ratio of the difference between the two is smaller than a preset threshold value.

Optionally, the relation between the aperture size l of the imaging units in the imaging unit array and the pixel size p of the imaging sensor satisfies l being larger than or equal to 2p.

Optionally, the main lens is a monolithic lens or a lens group; when the main lens is a monolithic lens, the gating window layer is positioned on the light-facing side of the main lens; when the main lens is a lens group, the gating window layer is positioned on the light-facing side of the main lens or between any two lens layers of the main lens.

In a second aspect, an embodiment of the present application provides a light field acquisition device, including any one of the light field acquisition structures, an eye positioning unit, and a gating area control unit;

the human eye positioning unit is configured to determine a human eye position;

the gating region control unit is configured to determine a region of a gating window in the light field acquisition structure in the gating window layer based on the human eye position.

Optionally, the light field acquisition device further comprises: a control processing circuit and a light field display unit;

the control processing circuit is respectively and electrically connected with the light field acquisition structure and the light field display unit, and is configured to acquire light field information acquired by an image sensor designated area in the light field acquisition structure according to a user instruction, process the light field information and send processed data to the light field display unit;

the light field display unit is configured to acquire the processed data and display according to the processed data.

In a third aspect, an embodiment of the present application provides a light field acquisition method based on a light field acquisition structure, where the light field acquisition structure includes: a gating window layer, a main lens, an imaging unit array, and an imaging sensor; the main lens, the imaging unit array and the imaging sensor are sequentially arranged at intervals;

the method comprises the following steps:

determining a plurality of target gating areas on the gating window layer, and sequentially configuring the target gating areas as gating windows; the gating window is a light-transmitting area on the gating window layer; the gating window is used for limiting the transmission area of light rays on the main lens;

acquiring light signals which are sensed by the image sensor, are incident to the transmission area and sequentially transmitted through the main lens and the imaging unit array during the period that any one of the target gating areas is configured as the gating window;

and determining light field information of a view point corresponding to the target gating region based on the light signal.

Optionally, the light field collection method based on the light field collection structure further comprises:

and fusing the light field information of the corresponding view point of each target gating region to obtain multi-view light field information.

The embodiment of the application has the beneficial effects that:

the light field acquisition structure provided by the embodiment of the application comprises a gating window layer, a main lens, an imaging unit array and an imaging sensor; the main lens, the imaging unit array and the imaging sensor are sequentially arranged at intervals; the gating window layer is provided with a gating window, and the gating window layer is configured to: restricting a transmission region of light on the main lens through the gating window; the gating window is a light-transmitting area on the gating window layer; the image sensor is configured to sense an optical signal incident to the transmission region and transmitted through the main lens, the imaging unit array in sequence.

Therefore, the light transmission area of the light on the main lens is restrained by arranging the gating window layer and configuring the light-transmitting gating window on the gating window layer, so that the incidence angle of an imaging light beam of an object point outside the light field acquisition structure is limited. When light rays incident from an object point pass through the gating window and are transmitted to the imaging unit array through the main lens, the light rays incident to each imaging unit in different directions are scattered to the degree of scattering on different pixels in a macro pixel in the imaging sensor, namely, the number of viewpoints acquired by the light field acquisition structure is reduced. On the premise that the total amount of the angle information and the space information which can be recorded by the imaging sensor is limited, compared with a light field acquisition structure without a gating window layer, the light field acquisition structure provided by the embodiment of the application reduces the number of acquired viewpoints, reduces the information amount of the angle information recorded by the image sensor, and improves the information amount of the space information which can be recorded by the image sensor, thereby improving the image resolution of a single viewpoint image.

Of course, it is not necessary for any one product or method of practicing the application to achieve all of the advantages set forth above at the same time.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the application, and other embodiments may be obtained according to these drawings to those skilled in the art.

FIG. 1 is a schematic diagram of a light field acquisition structure according to an embodiment of the present application;

FIG. 2 is a schematic diagram of another light field acquisition structure according to an embodiment of the present application;

FIG. 3 (a) is a schematic diagram of an aperture gating mode of a light field acquisition structure provided by an embodiment of the present application;

FIG. 3 (b) is a schematic diagram of an aperture full-open mode of the light field acquisition device according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a gating window provided by an embodiment of the present application;

FIG. 5 is a schematic flow chart of a light field acquisition method according to an embodiment of the present application;

fig. 6 (a) is a schematic diagram of a target gating region 1 configured as a gating window according to an embodiment of the present application;

FIG. 6 (b) is a schematic diagram of a target gating region 2 configured as a gating window provided by an embodiment of the present application;

fig. 6 (c) is a schematic diagram of the target gating region 3 configured as a gating window according to the embodiment of the present application;

fig. 7 is a schematic diagram of a light field acquisition method according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by the person skilled in the art based on the present application are included in the scope of protection of the present application.

In order to solve the problem of insufficient image resolution of a single-viewpoint image of an existing light field camera, the embodiment of the application provides a light field acquisition structure which is suitable for various devices with image acquisition functions, such as cameras, mobile phones, tablet computers and the like.

Fig. 1 is a schematic structural diagram of a light field acquisition structure according to an embodiment of the present application, and as shown in fig. 1, an embodiment of the present application provides a light field acquisition structure, including: a gate window layer 110, a main lens 120, an imaging unit array 130, and an imaging sensor 140; the main lens 120, the imaging unit array 130, and the imaging sensor 140 are sequentially disposed at intervals;

the gating window layer 110 is provided with a gating window 111, and the gating window layer 110 is configured to: restricting a transmission region of light on the main lens 120 through the gate window 111; the gating window 111 is a light-transmitting region on the gating window layer 110;

the image sensor 140 is configured to sense an optical signal incident to the transmission region and transmitted through the main lens 120, the imaging unit array 130 in sequence.

It should be appreciated that areas of the gating window layer 110 other than the gating window 111 are opaque.

Specifically, after the light propagates to the gating window layer 110, only the portion of the light propagating to the gating window 111 may propagate to the main lens 120 through the gating window 111, and be further transmitted to the imaging unit array 130 by the main lens 120. That is, the area covered on the main lens 120 by the light transmitted through the gate window 111 may be understood as a transmission area on the main lens 120.

The main lens 120, the imaging unit array 130, and the imaging sensor 140 are sequentially disposed at intervals, and the specific intervals between the main lens 120, the imaging unit array 130, and the imaging sensor 140 may be selected based on actual requirements, which is not limited in the embodiment of the present application.

As an example, the imaging unit array 130 may be disposed on the image plane of the main lens 120, and the imaging sensor 140 may be disposed on the image plane of the imaging unit array 130 to increase the angular sampling rate of the light field acquisition structure for light rays.

The main lens 120 may be a convex lens, and is configured to collect received light, form a viewpoint image on an image plane of the main lens 120, and further transmit the light transmitted through the main lens 120 to the imaging unit array 130.

The size, shape and focal length of the main lens 120 may be selected according to practical requirements, which are not limited in the embodiment of the present application.

The imaging unit array 130 specifically refers to an array formed by a plurality of imaging units, where the imaging unit array 130 is disposed between the main lens 120 and the image sensor 140, and each imaging unit covers pixels in a certain area on the image sensor 140, that is, light transmitted to each imaging unit is further transmitted to pixels in a corresponding area on the image sensor 140, and light signals are recorded by the pixels on the image sensor 140.

Pixels herein may be understood as photosensitive elements on the image sensor 140, and further, pixels covered by each imaging element on the image sensor 140 may be referred to as macro-pixels, and individual pixels contained within the macro-pixels may be referred to as meta-pixels or picture elements.

After the imaging unit array 130 further transmits the light to the image sensor 140, the view point on the image plane of the main lens 120 is further decomposed, that is, the light incident on the same imaging unit is dispersed to different pixels on the macro pixel based on different light propagation directions, so that the light field acquisition structure can record the light field information of multiple view points.

As one example, the imaging unit array 130 may be a pinhole array or a microlens array. The number and arrangement of the imaging units included in the imaging unit array 130 may be selected according to practical requirements, and the size, focal length, and number of each imaging unit are not limited in this embodiment of the present application.

The image sensor 140 may specifically include a plurality of pixels, each of which may sense an optical signal.

As an example, the image sensor 140 may be specifically a CIS (Complementary metal oxide semiconductor Image Sensor ), a CCD (Charge Coupled Device, charge coupled device) or a PIN (Photodiode) array, or other array type photosensor. By applying the light field acquisition structure provided by the embodiment of the application, light rays are restored by the light path reversibility principle based on the light signals sensed by the image sensor 140, so that light field information to be acquired can be obtained.

After the light is restored, the relative position of the pixel in the macro-pixel in the image sensor 140 relative to the center of the imaging unit can represent the angle information of the light, and the position of the imaging unit corresponding to the pixel in the image sensor 140 can represent the position information of the light. Specifically, if the plane of the image sensor 140 is represented by (u, v) and the plane of the imaging unit array 130 is represented by (s, t), the light field can be specifically expressed as a function L (u, v, s, t).

In addition, in some application scenes in which more viewpoints do not need to be acquired, only a few viewpoints are focused to meet the requirements. For example, when a light field image is observed in real time by a human eye, the human eye can observe only a few viewpoints due to the limitation of the pupil size of the human eye, and the real-time performance of observation and the priority of the image resolution of the viewpoints are higher than the number of the viewpoints in the scene.

Fig. 2 is a schematic diagram of another structure of a light field acquisition structure according to an embodiment of the present application, in which fig. 2 specifically shows an aperture size a of the through window 111 and a focal length F of the main lens 120 ₁ Aperture size l of imaging unit in imaging unit array 130, focal length F of imaging unit ₂ The pixel size p on the imaging sensor 140.

In one embodiment of the application, the aperture size A of the gating window 111, the focal length F of the main lens 120 ₁ Aperture size l of imaging unit in imaging unit array 130, focal length F of imaging unit ₂ The relationship between them satisfies: l/F ₂ And A/F ₁ The ratio of the difference between the two is smaller than a preset threshold value.

As mentioned above, each imaging unit in the imaging unit array 130 covers a certain number of pixels on the imaging sensor 140, so that the number of pixels covered by each imaging unit is as large as possible, and the pixels covered by different imaging units do not interfere with each other, so as to achieve maximum utilization of the pixels in the imaging sensor 140, and F-number matching needs to be performed on the main lens 120 and the imaging unit array 130.

The F-number, i.e. the ratio of the focal length of the imaging element to the aperture size, is in the embodiment of the application the F-number of the main lens 120 is specificallyThe ratio of the focal length of the main lens 120 to the aperture size of the gating window 111. If the F-number of the main lens is large, i.e. F ₁ A larger/a will result in the diameter of the unit image behind each imaging unit being smaller than the diameter of the imaging unit, and the pixels on the imaging sensor 140 are wasted if the F-number of the main lens is smaller, i.e. F ₁ A smaller/a may result in overlapping of the unit images behind each imaging unit, resulting in crosstalk. Therefore, when light field information is acquired by the light field acquisition structure, the main lens 120 and the imaging unit array 130 need to be subjected to F-number matching, and in particular, the l/F is needed ₂ And A/F ₁ The ratio of the difference values is smaller than a preset threshold value, and the size of the preset threshold value can be selected according to actual requirements.

In practical applications, the aperture size l of the imaging unit can be generally set at a×f ₂ /F ₁ Slightly enlarged on the basis of (a).

In one embodiment of the application, the light field acquisition structure further comprises: a gating window control unit;

the gating window control unit is configured to apply a first control signal to the gating window layer 110 for controlling the shape, position and/or aperture size of the gating window 111.

Specifically, the first control signal may be an electrical signal, and by applying the electrical signal to the gating window layer 110, light transmission or light-tightness of the material at a specific position in the gating window layer 110 may be adjusted. Accordingly, the shape, position, and/or aperture size of the gating window 111 may be determined according to the need, thereby determining a required first control signal, and configuring the gating window 111 to be in a light-transmitting state by applying the first control signal to the gating window layer 110.

As one example, the gating window layer 110 may be made of liquid crystal, electrochromic glass, electro-wetting ink, or the like.

In practical applications, different first control signals may be applied to the gating window layer 110 according to the viewpoint positions and the viewpoint numbers for imaging, so as to configure the gating windows 111 with different shapes, positions and/or aperture sizes.

As an example, in the process of applying the light field acquisition structure provided by the embodiment of the present application, the gating window 111 may be moved according to the viewpoint to be imaged, and at this time, the image position obtained by the imaging sensor 140 may also be moved, and the pixel position corresponding to each viewpoint may be obtained by using an off-line calibration manner.

As an example, the aperture size of the gating window 111 may be configured to be different sizes, so that the light field acquisition structure provided by the embodiment of the present application operates in different modes, and the operation mode of the light field acquisition structure is exemplarily described below with reference to fig. 3 (a) and 3 (b).

Fig. 3 (a) is a schematic diagram of an aperture gating mode of a light field acquisition structure according to an embodiment of the present application, as shown in fig. 3 (a), in the aperture gating mode, a partial area on a gating window layer 110 is configured as a gating window 111, after light emitted by an object point is transmitted to the gating window layer 110, only the light on the gating window 111 may be incident on a main lens 120, and transmitted sequentially through the main lens 120 and an imaging unit array 130, and finally transmitted to an imaging sensor 140.

In the mode of operation shown in fig. 3 (a), each macro-pixel on the imaging sensor 140 is able to sense a small number of light rays from different directions, and therefore, the light field acquisition structure can be used to acquire light field information for only a small number of viewpoints.

Fig. 3 (b) is a schematic diagram of an aperture full-open mode of the light field acquisition device according to the embodiment of the present application, as shown in fig. 3 (b), in the aperture full-open mode, the whole gating window layer 110 is in a light-transmitting state, and at this time, the light field acquisition structure according to the embodiment of the present application can be equivalent to a conventional integrated light field camera.

In the operation mode shown in fig. 3 (b), each macro-pixel on the imaging sensor 140 is capable of sensing a plurality of light rays from different directions, and thus, the light field acquisition structure can be used to acquire viewpoint information of a plurality of viewpoints, but in this operation mode, a large number of pixels on the imaging sensor 140 are used to record the viewpoint information, and thus, the image resolution of a single viewpoint image is insufficient.

It should be noted that when the aperture size of the gating window 111 is adjusted to enable the light field acquisition structure to operate in different modes of operation, in order to still satisfy the F-number matching condition between the main lens 120 and the imaging unit array 130, the aperture size of the imaging units in the imaging unit array 130 needs to be correspondingly adjusted.

In practical application, when the number of viewpoints focused by a user is small, but the requirement on image resolution is high, the light field acquisition structure can be configured into an aperture gating mode; when a user needs to record light field information of multiple viewpoints at the same time, for example, an application scene for acquiring the light field information aiming at a non-real-time scene, the light field acquisition structure can be configured into an aperture full-open mode.

In the embodiment of the application, the shape, the position and/or the aperture size of the gating window 111 can be flexibly configured by applying the first control signal on the gating window layer 110 based on the viewpoint position and the viewpoint number required to be imaged, and the light field acquisition structure has higher flexibility for different application requirements.

In one embodiment of the present application, the imaging unit array 130 is a liquid crystal lens array;

the light field acquisition structure further comprises: a lens control unit;

the lens control unit is configured to apply a second control signal to the liquid crystal lens array, the second control signal being used to control an aperture size of the liquid crystal lenses in the liquid crystal lens array.

Wherein the second control signal is specifically an electrical signal, and when the imaging unit array 130 is specifically a liquid crystal lens array, flexible configuration of the aperture size of the liquid crystal lens can be achieved by applying the second control signal to the liquid crystal lens array.

Therefore, in the embodiment of the present application, when the aperture size of the gating window 111 is adjusted according to the actual requirement, the F-number matching between the main lens 120 and the imaging unit array 130 can be completed by applying the second control signal to the imaging unit array 130, so that the light field acquisition structure has higher flexibility when facing different application requirements.

In one embodiment of the present application, the relationship between the aperture size l of the imaging cells in imaging cell array 130 and the pixel size p of imaging sensor 140 satisfies l+.2p.

As described above, light rays incident on a single imaging unit are dispersed to different picture elements in macro-pixels based on different light ray directions, i.e. different picture elements in each macro-pixel are used to record information at different viewpoints.

When the light field information acquired by the light field acquisition structure is actually applied, the premise that the human eye forms monocular focusing is that at least more than two viewpoint images enter pupils, so that a single imaging unit needs to be ensured to cover at least two pixels on the imaging sensor 140 when the light field information is acquired, namely, l is more than or equal to 2p, so that the actual requirement on the light field acquisition structure is met, and at the moment, each macro pixel at least comprises two pixels.

In one embodiment of the present application, the primary lens 120 is a monolithic lens or lens group; when the main lens 120 is a monolithic lens, the gating window layer 110 is located on the light-facing side of the main lens 120; when the main lens 120 is a lens group, the gating window layer 110 is located on the light-facing side of the main lens 120 or between any two lens layers of the main lens 120.

Specifically, the gating window layer 110 illustrated in fig. 1 is located on the light-facing side of the main lens 120.

In practical applications, the main lens 120 may be a lens group in order to ensure an imaging effect. For example, when the light field collecting device provided by the embodiment of the present application is applied to a camera, the main lens 120 may be a combined lens in the camera, that is, a lens group, in this case, besides disposing the gating window layer 110 on the light-facing side of the main lens 120, the gating window layer 110 may also be disposed between any two lens layers of the main lens 120, where the gating window layer 110 in both configurations can achieve a constraint effect on a transmission area of light on the main lens 120, and the gating window layer 110 is flexible in configuration in different imaging devices and has a wide application range of the light field collecting structure.

Based on the same inventive concept, the embodiment of the application also provides a light field acquisition device, which comprises the light field acquisition structure, the human eye positioning unit and the gating area control unit in any one of the embodiments;

the human eye positioning unit is configured to determine a human eye position;

the gating region control unit is configured to determine the region of the gating window 111 in the gating window layer 110 in the light field acquisition structure based on the human eye position.

In the embodiment of the present application, before the gating window 111 is configured on the gating window layer 110, the human eye position of the user may be determined by the human eye positioning unit.

As an example, the human eye positioning unit may in particular be an eye tracking camera.

After determining the human eye position, the region of the gating window 111 in the gating window layer 110 in the light field acquisition structure can be determined based on the human eye position, the gating window 111 is configured in the corresponding region on the gating window layer 110, and then the light field acquisition is performed based on the light field acquisition device with the configured gating window 111.

Specifically, for a single user, two gating windows 111 may be configured on the gating window layer 110 according to the located human eye position.

Fig. 4 is a schematic diagram of a gating window provided in an embodiment of the present application, as shown in fig. 4, a eye tracking (eyeTracking) camera is used to locate a position of a user's eyes, after the position of the eyes is determined, two areas corresponding to the position of the eyes on the gating window layer 110 are respectively configured as gating windows 111, and a distance between the two gating windows 111 is matched with a pupil distance of the user.

When the eye positioning unit is specifically an eye tracking camera, and the eye tracking camera positions the eyes of the user, the gating area control unit may also adjust the area of the gating window 111 on the gating window layer 110 based on the eye tracking camera.

In practical applications, since different people have different interpupillaries, if the gating window 111 is configured in a fixed area on the gating window layer 110, the difference of imaging effects when the light field acquisition device is used by different users is large. Therefore, the light field acquisition device provided by the embodiment of the application determines the position of the human eye through the human eye positioning unit, and the gating window 111 on the gating window layer 110 is configured based on the position of the human eye, so that the light field acquisition device can be ensured to have better effects when being applied to different users, and at the moment, the view point acquired by the light field acquisition device is closer to the real viewing effect of the human eye.

As an example, the light field acquisition device provided by the embodiment of the present application may be specifically applied to a head-mounted MR (Mixed Reality) device.

In one embodiment of the application, the light field acquisition device further comprises: a control processing circuit and a light field display unit;

the control processing circuit is respectively and electrically connected with the light field acquisition structure and the light field display unit, and is configured to acquire light field information acquired by the designated area of the image sensor 140 in the light field acquisition structure according to a user instruction, process the light field information and send the processed data to the light field display unit;

The light field display unit may specifically be a light field display.

As described above, based on the principle of reversibility of the light path, the position information and the angle information of the light sensed by each pixel on the imaging sensor 140 can be restored, and in addition, the intensity information of the light is recorded on the pixel of the imaging sensor 140, so that the light field acquisition structure records the position, the angle and the intensity information of the light at the same time. Therefore, by processing the information acquired by the imaging sensor 140 and displaying it by the light field display unit, an external real image can be reproduced.

Specifically, the process of processing the light field information by the control processing circuit can be understood as refocusing, that is, by moving the image plane position for objects with different object distances, performing integral operation on the light field information acquired by the light field acquisition structure, and then obtaining a clear image of the object on the image plane corresponding to the object. For a specific procedure of refocusing, reference may be made to the content in the related art.

Based on the same inventive concept, the embodiment of the application also provides a light field acquisition method based on a light field acquisition structure, wherein the light field acquisition structure comprises: a gate window layer 110, a main lens 120, an imaging unit array 130, and an imaging sensor 140; the main lens 120, the imaging unit array 130, and the imaging sensor 140 are sequentially disposed at intervals;

fig. 5 is a schematic flow chart of a light field acquisition method provided by an embodiment of the present application, referring to fig. 5, the light field acquisition method includes:

step S501: determining a plurality of target gating areas on a gating window layer, and sequentially configuring the target gating areas as gating windows; the gating window is a light-transmitting area on the gating window layer; the gating window is used to restrict the transmission area of light on the main lens.

Step S502: during the period that any target gating region is configured as a gating window, light signals which are sensed by the image sensor, are incident to the transmission region and sequentially transmitted through the main lens and the imaging unit array are acquired.

Step S503: based on the light signals, light field information of a view point corresponding to the target gating region is determined.

The specific content of the light field acquisition structure may refer to the description in any of the foregoing embodiments, and will not be described herein.

In practical applications, a user may need to acquire light field information of a plurality of different viewpoints, and for a static scene, in order to improve the image resolution of a single viewpoint image that can be obtained, a viewpoint that needs to be imaged may be determined on the gating window layer 110 to determine a plurality of target gating areas, and the target gating areas are scanned, that is, the target gating areas are sequentially configured as the gating windows 111.

During any one of the target gate regions configured as the gate window 111, the image sensor 140 may sense a light signal incident to the transmission region of the main lens 120 and transmitted through the main lens 120, the imaging unit array 130 in sequence.

Specifically, by configuring different target gating regions as gating windows 111, imaging sensor 140 may specifically sense light emitted by the same object point in different directions for the light emitted by the object point. Therefore, it can be also understood that during the period that any target gating area is configured as the gating window 111, the light field acquisition structure can acquire light field information corresponding to one sub-viewpoint set in the viewpoints to be acquired, and by scanning the target gating areas, light field information corresponding to different sub-viewpoint sets can be acquired, so that light field information of multiple viewpoints can be obtained.

Therefore, in practical application, multiple target gating areas may be determined on the gating window layer 110 according to the viewpoint to be imaged, where each target gating area corresponds to one sub-viewpoint set.

The specific position of the target gating region on the gating window layer 110 may be selected according to actual requirements, which is not limited in the embodiment of the present application.

As one example, in order to ensure continuity of viewing angle and to avoid wasting of pixels caused by recording redundant information by the imaging sensor 140, a plurality of regions continuous with each other and non-overlapping with each other on the gate window layer may be regarded as target gate regions.

Fig. 6 (a) -6 (c) respectively show schematic diagrams when target gating regions at different positions are configured as gating windows, and an embodiment of the present application is described below by way of example with reference to fig. 6 (a) -6 (c).

Referring to fig. 6 (a), the target gating region 1 is configured as a gating window 111, and for an object point illustrated in the figure, the light field acquisition structure may specifically acquire light field information of light rays of the object point between light ray directions of light ray a and light ray b.

Referring to fig. 6 (b), the target gating region 2 is configured as a gating window 111, and for an object point illustrated in the figure, the light field acquisition structure may specifically acquire light field information of light rays of the object point between the light ray directions of the light rays b and c.

Referring to fig. 6 (c), the target gating region 3 is configured as a gating window 111, and for an object point illustrated in the figure, the light field acquisition structure may specifically acquire light field information of light rays of the object point between the light ray directions of the light ray c and the light ray d.

Therefore, by configuring the target gating regions 1-3 as the gating windows 111 respectively, the light field acquisition structure can acquire light field information in different direction ranges, so that the light field information of the light rays between the light ray directions of the light ray a and the light ray d can be obtained by scanning the target gating regions 1-3, that is, viewpoint information corresponding to more viewpoints can be obtained, and the obtained light field image based on each viewpoint has higher image resolution.

In the embodiment of the application, the plurality of target gating areas are determined on the gating window layer, and the target gating areas are sequentially configured as the gating windows, so that the light field information of different sub-viewpoint sets can be acquired through the light field acquisition structure, and the requirement of a user for observing more viewpoints can be met on the basis of ensuring the image resolution of a single-viewpoint image.

In one embodiment of the application, the light field acquisition method further comprises:

As described above, during the period that each gating region is configured as a gating window, light field information of one sub-viewpoint set can be obtained, so that light field information of the corresponding viewpoint of each target gating region is fused, and multi-viewpoint light field information can be obtained. The light field image obtained on the basis has the characteristics of large specific viewpoint range, high viewpoint density and high image resolution.

Fig. 7 is a schematic diagram of a light field acquisition method according to an embodiment of the present application, and the following description will further explain the embodiment of the present application with reference to fig. 7, where, as shown in fig. 7, the light field acquisition method may specifically include the following steps:

step S701: the gating window is altered.

Specifically, after light field acquisition based on the currently configured gating window, the next non-scanned target gating region may be configured as a gating window.

Step S702: light field images are acquired.

Specifically, after the change of the gating window is completed, a light field image is acquired through a light field acquisition structure.

Step S703: whether all window collection is completed is judged, if yes, step S704 is executed, and if no, step S701 is returned.

Specifically, it is determined whether all the target gating areas are scanned, if yes, step S704 is executed, and if not, step S701 is returned to configure the next target gating area as a gating window until all the target gating areas are scanned.

Step S704: all light field images are fused.

Step S705: and outputting the high-resolution light field image.

Specifically, after all the target gating areas are scanned, the light field images corresponding to each target gating area are fused, so that a super-resolution light field image can be obtained, the light field image has a large number of viewpoints, and the image resolution of a single viewpoint is high.

In the embodiment of the application, the multi-viewpoint light field information is obtained by fusing the light field information of the viewpoints corresponding to each target gating region, and the light field image obtained on the basis of the multi-viewpoint light field information contains the light field information of more viewpoints besides the higher image resolution of a single viewpoint, so that the requirements of users on the viewpoint number and the image resolution can be simultaneously met.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the light field acquisition device and the light field acquisition method, since the light field acquisition device and the light field acquisition method are basically similar to the light field acquisition structure embodiment, the description is simpler, and the relevant points are only referred to the part of the description of the method embodiment.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A light field acquisition structure, comprising: a gating window layer, a main lens, an imaging unit array, and an imaging sensor; the main lens, the imaging unit array and the imaging sensor are sequentially arranged at intervals;

2. The light field acquisition structure of claim 1, wherein the light field acquisition structure further comprises: a gating window control unit;

3. The light field acquisition structure of claim 1, wherein the array of imaging units is an array of liquid crystal lenses;

the light field acquisition structure further comprises: a lens control unit;

4. The light field acquisition structure of claim 1, wherein an aperture size a of the gating window, a focal length F of the main lens ₁ An aperture size l of an imaging unit in the imaging unit array, a focal length F of the imaging unit ₂ The relationship between them satisfies: l/F ₂ And A/F ₁ The ratio of the difference between the two is smaller than a preset threshold value.

5. The light field acquisition structure of claim 1 wherein the relationship between the aperture size l of an imaging unit in the array of imaging units and the pixel size p of the imaging sensor satisfies l+.2p.

6. The light field acquisition structure of claim 1 wherein the primary lens is a monolithic lens or a lens group; when the main lens is a monolithic lens, the gating window layer is positioned on the light-facing side of the main lens; when the main lens is a lens group, the gating window layer is positioned on the light-facing side of the main lens or between any two lens layers of the main lens.

7. A light field acquisition device, characterized by comprising the light field acquisition structure of any one of claims 1-6, an eye positioning unit, a gating area control unit;

the human eye positioning unit is configured to determine a human eye position;

8. The light field acquisition device of claim 7, further comprising: a control processing circuit and a light field display unit;

9. A light field acquisition method based on a light field acquisition structure, the light field acquisition structure comprising: a gating window layer, a main lens, an imaging unit array, and an imaging sensor; the main lens, the imaging unit array and the imaging sensor are sequentially arranged at intervals;

the method comprises the following steps:

10. The method as recited in claim 9, further comprising: