WO2020244273A1

WO2020244273A1 - Dual camera three-dimensional stereoscopic imaging system and processing method

Info

Publication number: WO2020244273A1
Application number: PCT/CN2020/079099
Authority: WO
Inventors: 李应樵; 陈增源
Original assignee: 万维科研有限公司
Priority date: 2019-06-04
Filing date: 2020-03-13
Publication date: 2020-12-10
Also published as: CN112040214A

Abstract

Disclosed by the present invention are a dual camera three-dimensional stereoscopic imaging system and a processing method; a light field image capture part comprises a first imaging part, a first camera lens and a second camera or camera lens; the first camera lens and the second camera or camera lens are located at a rear portion and a front portion of a lens part respectively, an entrance pupil plane and a matching device are placed therebetween, an internal reflection unit is formed between the first camera lens, the entrance pupil plane and the matching device, and a high-resolution image capture part further comprises a second imaging part and a third camera lens; the light field image capture part and the high-resolution image capture part are configured so that the third camera lens obtains a second image which is consistent with the vertical direction of a front view in a plurality of secondary images, and at the same time, the plurality of secondary images and the second image are outputted. In addition to obtaining accurate depth information, the present invention may also obtain high-resolution video to analyze three-dimensional imaging.

Description

Dual-camera three-dimensional imaging system and processing method

Technical field

The invention belongs to the field of stereo imaging, and particularly relates to a dual-camera three-dimensional imaging system and processing method based on light field technology.

Background technique

In the prior art, there are a variety of design solutions for cameras used for shooting three-dimensional images and videos. The most common solution is to separate two camera modules of the same specification at a certain distance, such as about 60 to 65 mm, in a linear arrangement to simulate the principle of stereo vision of the human eye. The image sensors of the two camera modules will separately record the two-dimensional images or videos taken by each, and the two two-dimensional images or two two-dimensional videos obtained by software processing can be used to establish a depth map, which is then converted into a three-dimensional image or three-dimensional video. Another solution is to directly use a stereo camera to shoot. There are two image sensors in the main body of the stereo camera to record two-dimensional images or two-dimensional videos from the two lens groups of the camera lens. Then the system and software attached to the camera will combine the two two-dimensional images into a three-dimensional image, or combine two segments Two-dimensional video is synthesized into three-dimensional video. However, these two solutions may have the quality of the three-dimensional image and video generated by the asynchronous two-dimensional image or video, or the external factors such as the environmental lighting conditions.

More advanced imaging equipment, such as light field camera, also known as plenoptic camera, uses microlens array lenses to capture the light field image of the scene at a time, and the depth information of the scene can be extracted through calculation to create a depth map And convert the two-dimensional image into a three-dimensional image. However, the main disadvantages of this kind of light field camera equipment are that the image resolution will drop significantly, the parallax angle is small, and it is not suitable for shooting video. The latest design is to add a reflecting unit to capture the multi-angle image of the target object. Because the parallax angle is large, it can produce a clearer depth map and three-dimensional image after processing. It is also suitable for shooting video. Attempts still failed to solve the problem of resolution drop.

Summary of the invention

The purpose of the present invention is to provide a dual-camera three-dimensional imaging system and processing method for improving the resolution of three-dimensional video. In many fields, such as medical treatment, biotechnology research, industrial equipment manufacturing, semiconductor product quality verification, etc., the imaging system has a wide range of applications. In addition to obtaining accurate depth information, the present invention can also obtain high-quality video and three-dimensional image analysis.

The present invention provides a dual-camera three-dimensional imaging system, which is characterized in that it comprises: a light field imaging part for obtaining a first image and a high resolution imaging part for obtaining a second image; wherein the light field imaging part includes the first imaging part. Part, the first camera lens and the second camera or camera lens; the first camera lens and the second camera or camera lens are respectively located at the rear and front of the lens part, and an entrance pupil plane and matching device are placed between the two, The entrance pupil plane and the matching device can be adapted to the different focal lengths of the second camera or camera lens, and an internal reflection unit is formed between the first camera lens and the entrance pupil plane and the matching device. The captured first image is decomposed and refracted into a plurality of secondary images with different angular offsets, the high-resolution imaging part further includes a second imaging part and a third camera lens, and at least one capable of adjusting the first The central axis adjustment device of the camera lens and the second camera or the double lens of the camera lens and the single lens of the third camera lens, the central axis adjusting device keeps the axes of the double lens and the single lens parallel; configuring the light The field imaging part and the high-resolution imaging part enable the third camera lens to obtain a second image consistent with the vertical direction of the front view among the plurality of secondary images, and simultaneously output the plurality of secondary images And the second image.

In one aspect of the present invention, the distance between the light field imaging part and the high-resolution imaging part is as close as possible, and their centers are located on the same vertical plane.

In an aspect of the present invention, the angular offset range of the plurality of secondary images with different angular offsets is 10-20 degrees.

In an aspect of the present invention, the angular offset of the front view in the plurality of secondary images is 0 degree. The first imaging part further includes a first image sensor and a fly-eye lens that captures a first image; the fly-eye lens transmits the captured first image to the first image sensor; and the second imaging part further includes Second image sensor; the second image obtained by the third camera lens is transmitted to the second image sensor.

In one aspect of the present invention, the fly-eye lens is a plurality of micro lens arrays, and the radius, thickness, and array pitch of each micro lens are related to the size of the first image sensor.

In one aspect of the present invention, the aperture and focal length of the first camera lens and the second camera or camera lens are adjustable, and the second camera or camera lens and the third camera lens are replaceable lenses, and The aperture of the second camera or camera lens is larger than the size of the internal reflection unit.

In one aspect of the present invention, the entrance pupil plane and the matching device is a pupil lens, the diameter of the pupil lens is larger than the diameter of the internal reflection unit, and the incident light of the light field image is allowed to proceed in the internal reflection unit refraction.

In one aspect of the present invention, each of the secondary images has subtle differences in the scene, and the size of the internal reflection unit and the focal length of each secondary image are calculated based on the following equations (1) and (2):

Wherein, FOV is the field of view of the second camera or camera lens;

n is the refractive index of the internal reflection unit;

r is the number of internal reflections;

Z is the size of the internal reflection unit;

f _lens is the focal length of the second camera or camera lens;

f _sub is the focal length of the secondary image.

The present invention also provides a dual-camera three-dimensional imaging processing method, the steps of which are: obtaining the original depth map data of the first image through the light field camera part; correcting the original depth map data; using edge-oriented or directional rendering The method is to obtain a high-resolution depth map generated by interpolation; at the same time, a second image is obtained by using a high-resolution camera part, and a data model is used to combine the second image as reference data on the original depth map data of the first image. Correct until the best interpolated high-resolution depth map is obtained.

The three-dimensional imaging system and processing method provided by the present invention can provide higher-resolution two-dimensional and three-dimensional video, and at the same time, compared with the light field camera of the high-resolution image sensor, the cost increase is very limited; in addition, due to the The system does not affect the function of the light field camera part, so the information obtained by the light field camera itself can still be used to calculate the object depth and build a depth map.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some examples of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

Figure 1 is a perspective view of the three-dimensional imaging system of the present invention.

Figure 2 is a structural diagram of the three-dimensional imaging system of the present invention.

FIG. 3 is a schematic diagram of the first image 120 obtained by the three-dimensional imaging system of the present invention.

FIG. 4 is a schematic diagram after the three-dimensional imaging system according to the present invention performs normalization processing on the obtained first image 120.

FIG. 5 is a flowchart of processing the second image 130 by the three-dimensional imaging system of the present invention.

Fig. 6 is a flow chart of obtaining a target image by the three-dimensional imaging system of the present invention.

Detailed ways

The specific embodiments of the present invention will now be described with reference to the corresponding drawings. However, the present invention can be implemented in many different forms and should not be construed as being limited to the embodiments shown here. These embodiments are provided only for the detailed and comprehensiveness of the present invention, so that the scope of the present invention can be fully described to those skilled in the art. The wording used in the detailed description of the embodiments illustrated in the drawings should not limit the present invention.

Figure 1 is a perspective view of the three-dimensional imaging system of the present invention. The three-dimensional imaging system of the present invention is composed of a light field imaging part 100 that obtains a first image 120 (not shown in FIG. 1) and a high-resolution imaging part 140 that obtains a second image 130 (not shown in FIG. 1), wherein The light field camera part 100 can adopt the light field camera in Chinese patent application 201711080588.6, which includes a first imaging part 110, a first camera lens 101 and a second camera or camera lens 103, where the first camera lens 101 is a rear camera lens; It has adjustable aperture and focal length. The second camera or camera lens 103 is a front camera or camera lens, and the front and rear cameras or camera lenses can adjust the focal length of the camera. Between the first camera lens 101 and the second camera or camera lens 103 is the entrance pupil plane and the matching device 109, the entrance pupil plane and the matching device 109 may be a pupil lens, and between the pupil lens 109 and the first camera lens 101 is an internal reflection unit 102. The high-resolution imaging part 140 and the light field imaging part 100 are integrated and fixed together. The high-resolution imaging part 140 includes a second imaging part 116. The third camera lens of the high-resolution imaging part 140 is connected through the central axis adjustment device 118. The lens center axis 112a (see FIG. 2) of 117 and the lens center axis 112b (see FIG. 2) of the first camera lens 101 and the second camera lens 103 in the light field imaging part 100 are kept parallel.

Figure 2 is a structural diagram of the dual-camera three-dimensional imaging system of the present invention. Among them, the light field imaging part 100 of the three-dimensional imaging system includes a first imaging part 110 and a lens part 111, wherein the first imaging part 110 includes a first image sensor 104; a fly-eye lens 105; wherein the first image sensor 104 uses a higher imaging quality High image sensor; fly-eye lens 105 is formed by a combination of a series of small lenses, capturing information of a certain image from different angles, such as light field image information, so as to strip out three-dimensional information to identify specific objects. The fly-eye lens 105 is composed of a micro lens array and is designed to not only capture a light field image, but also generate a depth map. In addition, the fly-eye lens 105 serves the first image sensor 104, so it is related to the parameters of the first image sensor 104. For example, each micro lens parameter of the fly-eye lens 105 has a radius of 0.5 millimeters, a thickness of 0.9 micrometers, and the array pitch of each micro lens is 60 micrometers. With respect to the first image sensor 104, the size of the fly-eye lens is retractable. In one embodiment, the size of the C-type image sensor using the advanced photography system is 25 mm×17 mm; and in another embodiment, the size of the full-frame image sensor is 37 mm×25 mm.

The lens part 111 is detachably connected with the first imaging part 110. The pupil lens 109 may be a single lens, which has a condensing effect and can compress the information received by the second camera or camera lens 103. An imaging process is performed at the second camera or camera lens 103, and as the second camera or camera lens 103 is replaced or replaced, the imaging angle is different. The first camera lens 101 is a short-focus lens or a macro lens, which is fixed on a housing (not shown in FIG. 2). The design of the first camera lens 101 determines the size of the imaging system of the present invention. A secondary imaging process is performed at the first camera lens 101. The entrance pupil plane and the matching device 109 are designed to correct light rays. Between the entrance pupil plane and the matching device 109 and the first lens 101 is an internal reflection unit 102; the internal reflection unit 102 decomposes and reflects the image to be taken into a multi-angle image with independent secondary images with different angular offsets. The internal reflection unit 102 is designed to provide multiple virtual images at different viewing angles. The size and ratio of the internal reflection unit 102 are the determining factors of the number of reflections and the reflection image ratio, and images of different angles are produced. The secondary image produced by each reflection has a subtle difference in the scene, and the target image has a slight offset. The size of the internal reflection unit 102 and the focal length of each secondary image can be calculated based on the following equations (1) and (2):

Wherein, FOV is the field of view of the second camera or camera lens;

n is the refractive index of the internal reflection unit;

r is the number of internal reflections;

X, Y, Z are the dimensions of the internal reflection unit, which are width, height, and length respectively;

f _lens is the focal length of the second camera or camera lens;

f _sub is the focal length of the secondary image.

The size of the internal reflection unit 102 can be the same as the size of the first image sensor 104, and in one embodiment, it can be 24 mm (width) x 36 mm (height) x 95 mm (length), that is to say the ratio of the unit It is about 2:3:8. The pupil lens 109 is used to match the size of the secondary image with the size of the internal reflection unit 102, and to perform reflection in the internal reflection unit 102 correctly. To achieve this, the diameter of the pupil lens 109 should be larger than the internal reflection unit 102. In one of the embodiments, the pupil lens 109 has a diameter of approximately 50 mm and a focal length of 50 mm. As long as the aperture of the second camera or camera lens 103 is larger than the size of the internal reflection unit 102, the second camera or camera lens 103 is designed to be able to be replaced by any camera or camera lens.

The high-resolution imaging section 140 includes a second imaging section 116, a second image sensor 119, and a third camera lens 117. The adjusting device 118 for adjusting the central axis of the dual lens of the first camera lens 101 and the second camera or the camera lens 103 and the single lens of the third camera lens 117 is located outside the high-resolution imaging part 140 and is independent of light field imaging The part 100 and the high-resolution imaging part 140 can make the axis 112b of the first camera lens 101, the second camera or the camera lens 103 parallel to the axis 112a of the third camera lens 117 by adjusting the adjustment device 118. The second image sensor 119 can be a sensor with the same or different specifications as the first image sensor 104, but the resolution of the second image sensor 119 should be at least 1/9 or more of the resolution of the first camera sensor 104 to achieve higher cost. The purpose of inventing the light field video resolution of the 3D imaging system.

3 is a schematic diagram of the first image 120 obtained by the light field imaging part 100 of the three-dimensional imaging system of the present invention. The internal reflection unit 102 in the light field imaging part 100 decomposes the captured first image 120, that is, the light field image or video picture, and reflects it into multiple secondary images or video pictures with different angular offsets, such as 9 Two secondary images or video frames are acquired by the first image sensor 104 of the first imaging part 110 through a fly-eye lens. The secondary image ① in the middle of the 9 secondary images or video images is the front view of the scene, and the remaining 8 secondary images ②--⑨ or video images are secondary images or video images offset by a specific angle. On average, each image or video screen has a resolution of 1/9 or lower of the first image sensor 104. Before generating the scene depth map, 9 secondary images or video frames are segmented and each secondary image is preprocessed.

4 is a schematic diagram of the first image 120 obtained by the light field imaging part 100 of the three-dimensional imaging system according to the present invention after normalization processing. Normalize each secondary image through the following equation (3):

Among them, I _n (n = 1, 2,..., 9) represents the image before normalization; _I'n (n = 1, 2,..., 9) represents the image after normalization; mirror(I _m ,left,right,up ,down)(m=1,2,3,4) represents the image mirroring to the left, right, up and down; rotate(I _k ,π)(k=6,...,9) represents the image rotation.

After the normalization process, the offset of the scene in each secondary image can be identified, and each secondary image is an independent original compound eye image. In the next step, image processing techniques including but not limited to image noise removal are used for preprocessing, and then synthetic aperture technology is used for decoding, the light field information in the original image of the compound eye can be obtained, and digital refocusing technology can be used to generate the secondary focus image. Among them, the synthetic aperture image can be digitally refocused using the following principles:

L′(u,v,x′,y′)=L(u,v,kx′+(1-k)u,ky′+(1-k)v) (6)

I′(x′,y′)=∫∫L(u,v,kx′+(1-k)u,ky′+(1-k)v)dudv (7) where I and I'represent one time And the coordinate system of the secondary imaging surface;

L and L'represent the energy of the primary and secondary imaging surfaces.

After obtaining 9 quasi-focus secondary images, you can select two secondary images, such as secondary images ② and ③, and use the stereo matching algorithm to calculate the disparity map (Disparity Map), and then further establish the comparison Low-resolution scene depth map. Among them, the semi-global matching (Semi-Global Matching) algorithm is one of the commonly used binocular stereo vision matching algorithms, which provides good parallax effects and ideal calculation speed. By setting up a cost function related to the disparity map:

Among them, D represents the disparity map;

p and q are a certain pixel in the image;

C(p, D _p ) represents the cost value of the pixel when the disparity value of the current pixel is D _p ;

N _p represents the pixels adjacent to the pixel p, usually 8;

P1 and P2 are penalty coefficients. P1 applies to pixel p and its neighboring pixels with a difference of disparity equal to 1, and P2 applies to pixel p and its neighboring pixels with a difference of disparity greater than 1;

T[.] is a function. If the parameter in the function is true, it returns 1, otherwise it returns 0.

Calculate the minimum cost value when it is a certain disparity value for each pixel of the image, and accumulate the cost values in 8 directions, and the cumulative disparity value with the lowest value is equivalent to the final disparity value of the pixel. After calculating each pixel one by one, the entire disparity map can be obtained.

The final conversion between the disparity value and the depth value can use the following formula:

Among them, d _p represents the depth value of a certain pixel;

f is the normalized focal length;

b is the baseline distance between the two secondary images;

D _p represents the disparity value of the current pixel.

The second image 130 obtained by the second imaging part 116 is a 2D image or a video screen, which can completely reflect the information of the subject, so the resolution of the second image will not be reduced, and for the same reason, the second image is not required. 130 for standardization and refocusing.

Taking the first image sensor 104 and the second image sensor 119 both adopting 4K image sensors as an example, that is, the first and second image sensors both have 3840×2160 pixels. Assuming that the above 9 secondary images or video images have the same size, on average, each secondary image has only 1/9 or lower resolution of the sensor. For example, the resolution of each secondary image is 1280x720 pixels. If the scene depth map is directly generated And light field video, its resolution will also be limited by 1280x720 pixels. Therefore, first use the secondary image ①－⑨ to establish a 1280x720 scene depth map, and then refer to the 3840x2160 pixel high-resolution second image 130 obtained by the second image sensor 119, and use the edge-directed interpolation algorithm to improve the depth map. The resolution is up to 3840x2160. The formula used for edge-directed interpolation is as follows:

Among them, m and n are low-resolution and high-resolution image grids before and after interpolation;

y[n] represents the depth map generated after interpolation;

x[m] and

Represents the original depth map and the corrected map;

Represents the reference data of the second image 130;

S and R respectively represent the data model of the second image 130 and the operator of the edge-directed rendering step;

λ is the gain of the correction process;

k is the iteration index.

Due to the high-resolution second image 130 as a basis, the accuracy of the correction step of the interpolation calculation is sufficient to meet the needs of generating a 3D image. Then the high-resolution second image 130, that is, a 2D image or video image, is combined with the increased resolution depth map to generate a high-resolution 2D+Z 3D image or video format and output to the display, which can greatly improve the light field image or The resolution of the light field video is up to 3840x2160 pixels.

Fig. 5 is a flowchart of the three-dimensional imaging system of the present invention processing a target image. In step 501, the original depth map data of the first image 120 is obtained through the first imaging part 100; in step 502, the original depth map data is corrected; in step 503, an edge-oriented or directional rendering method is used, and in step 504 Obtain a high-resolution depth map generated by interpolation; where in step 505, the data model of the second image is used to perform reference data on the second image obtained in step 506; and the original depth map data of the first image 120 Correct until the best interpolated high-resolution depth map is obtained.

Fig. 6 is a flow chart of obtaining a target image by the three-dimensional imaging system of the present invention. In step 601, the first image sensor 104 acquires a first image 120 containing 9 secondary images or video frames; in step 602, the 9 secondary images or video frames are divided and each secondary image is standardized In step 603, perform image noise removal processing on each secondary image; in step 604, use synthetic aperture technology to decode the light field information acquired by the 9 secondary images, and then use digital refocusing technology to generate an in-focus image; in step 605. Use 9 in-focus secondary images to establish a lower-resolution scene depth map; combine the second image of the high-resolution 2D image or video screen obtained by the second image sensor obtained in step 608; in step 606, referring to the second image, use an edge-oriented or edge-oriented interpolation algorithm to increase the resolution of the depth map; in step 607, combine the depth map with the increased resolution and the second image to generate a high-resolution 3D image or video .

The above descriptions are only used to illustrate the technical solutions of the present invention, and any person skilled in the art can modify and change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope of the claims. The present invention has been described above in conjunction with examples. However, other embodiments than the above-mentioned embodiments within the scope of the disclosure of the present invention are equally feasible. The different features and steps of the present invention can be combined in methods other than those described. The scope of the present invention is limited only by the appended claims. More generally, a person of ordinary skill in the art can easily understand that all the parameters, dimensions, materials and configurations described herein are for demonstration purposes and actual parameters, dimensions, materials and/or configurations will depend on the specific application or the present invention Teach the application used.

Claims

A dual-camera three-dimensional imaging system, characterized in that it comprises:

The light field imaging part for obtaining the first image and the high-resolution imaging part for obtaining the second image;

The light field photographing part includes a first imaging part, a first camera lens and a second camera or camera lens; the first camera lens and the second camera or camera lens are respectively located at the rear and front of the lens part, two An entrance pupil plane and matching device is placed in the middle, and the entrance pupil plane and matching device can be adapted to the different focal lengths of the second camera or camera lens, forming an interior between the first camera lens and the entrance pupil plane and the matching device A reflection unit for decomposing and refracting the captured first image into a plurality of secondary images with different angular offsets,

The high-resolution camera part also includes a second imaging part and a third camera lens, and at least one middle lens capable of adjusting the first camera lens and the second camera or the double lens of the camera lens and the single lens of the third camera lens. An axis adjustment device, the central axis adjustment device keeps the axes of the dual lens and the single lens parallel;

The light field imaging part and the high-resolution imaging part are configured so that the third camera lens obtains a second image consistent with the vertical direction of the front view among the plurality of secondary images, and simultaneously outputs the plurality of images. A secondary image and the second image.
The system according to claim 1, wherein the distance between the light field camera part and the high-resolution camera part is as close as possible, and their centers are located on the same vertical plane.
The system of claim 1, wherein the angular offset range of the plurality of secondary images with different angular offsets is 10-20 degrees.
The system of claim 3, wherein the angular offset of the front view in the plurality of secondary images is 0 degrees.
The system according to claims 1-4, wherein:

The first imaging part further includes a first image sensor and a fly-eye lens that captures a first image; the fly-eye lens transmits the captured first image to the first image sensor; and

The second imaging part further includes a second image sensor; the second image obtained by the third camera lens is transmitted to the second image sensor.
The system of claim 5, wherein:

The fly-eye lens is a plurality of micro lens arrays, and the radius, thickness, and array pitch of each micro lens are related to the size of the first image sensor.
The system according to any one of claims 1-4, 6, characterized in that:

The aperture and focal length of the first camera lens and the second camera or camera lens are adjustable, the second camera or camera lens and the third camera lens are replaceable lenses, and the second camera or The aperture of the camera lens is larger than the size of the internal reflection unit.
The system according to any one of claims 1-4, 6, characterized in that:

The entrance pupil plane and the matching device are a pupil lens, the diameter of the pupil lens is larger than the diameter of the internal reflection unit and allows the incident light of the light field image to be refracted in the internal reflection unit.
The system according to any one of claims 1-4, 6, characterized in that:

Each of the secondary images has a subtle difference in the scene, and the size of the internal reflection unit and the focal length of each secondary image are calculated based on the following equations (1) and (2):

Wherein, FOV is the field of view of the second camera or camera lens;

n is the refractive index of the internal reflection unit;

r is the number of internal reflections;

Z is the size of the internal reflection unit;

f lens is the focal length of the second camera or camera lens;

f sub is the focal length of the secondary image.
A processing method for dual-camera three-dimensional imaging, the steps are:

Obtain the original depth map data of the first image through the light field camera part;

Correcting the original depth map data;

Use edge-oriented or directional rendering methods to obtain high-resolution depth maps generated by interpolation;

At the same time, use the high-resolution camera part to obtain the second image, and use the data model to modify the original depth map data of the first image in combination with the second image as the reference data until the best high-resolution interpolation generated Rate depth map.
The processing method according to claim 10, wherein:

The light field camera part includes a first imaging part, a first camera lens and a second camera or camera lens; the first camera lens and the second camera or camera lens are respectively located at the rear and front of the lens part, and An entrance pupil plane and matching device is placed in the middle, and the entrance pupil plane and matching device can be adapted to the different focal lengths of the second camera or camera lens, forming an interior between the first camera lens and the entrance pupil plane and the matching device A reflection unit for decomposing and refracting the captured first image into a plurality of secondary images with different angular offsets, the high-resolution imaging part further includes a second imaging part and a third camera Lens, and at least one central axis adjusting device capable of adjusting the first camera lens and the second camera lens or the double lens of the camera lens and the single lens of the third camera lens. The central axis adjusting device makes the double lens and the The axis of the single lens is kept parallel; the light field imaging part and the high-resolution imaging part are arranged so that the third camera lens obtains a second image consistent with the vertical direction of the front view among the plurality of secondary images , And simultaneously output the plurality of secondary images and the second image.
The processing method according to claim 11, wherein:

The first image, which is 9 secondary images or video images acquired by the first image sensor;

Divide the 9 secondary images or video pictures and perform standardized processing on each secondary image;

Perform image noise removal processing on each secondary image; use synthetic aperture technology to decode the light field information obtained from the 9 secondary images, and then use digital refocusing technology to generate an in-focus image;

Use 9 in-focus secondary images to establish a lower resolution scene depth map; in combination with the second image, it is a high-resolution 2D image or video image obtained by the second image sensor;

With reference to the second image, an edge-oriented or edge-oriented interpolation algorithm is used to increase the resolution of the depth map; and a high-resolution 3D image or video is generated by combining the increased-resolution depth map and the second image.
The processing method according to claim 12, wherein each secondary image is normalized by the following equation:

Among them, I n (n = 1, 2,..., 9) represents the image before normalization; I'n (n = 1, 2,..., 9) represents the image after normalization; mirror(I m ,left,right,up ,down)(m=1,2,3,4) represents the image mirroring to the left, right, up and down; rotate(I k ,π)(k=6,...,9) represents the image rotation.
The processing method of claim 12, wherein the synthetic aperture image is digitally refocused using the following principle:

L′(u,v,x′,y′)=L(u,v,kx′+(1-k)u,ky′+(1-k)v)

(6)

I′(x′,y′)=∫∫L(u,v,kx′+(1-k)u,ky′+(1-k)v)dudv

(7)

Among them, I and I'represent the coordinate system of the primary and secondary imaging planes;

L and L'represent the energy of the primary and secondary imaging surfaces.
The processing method according to claim 12, wherein a disparity map (Disparity Map) is calculated by using a stereo matching (Stereo Matching) algorithm, and then a lower resolution scene depth map is further established.
15. The processing method of claim 15, wherein the binocular stereo vision matching algorithm is a semi-global matching (Semi-Global Matching) algorithm, by establishing the following cost functions related to the disparity map:

Among them, D represents the disparity map;

p and q are a certain pixel in the image;

C(p, D p ) represents the cost value of the pixel when the disparity value of the current pixel is D p ;

N p represents the pixels adjacent to the pixel p, usually 8;

P1 and P2 are penalty coefficients. P1 applies to pixel p and its neighboring pixels with a difference of disparity equal to 1, and P2 applies to pixel p and its neighboring pixels with a difference of disparity greater than 1;

T[.] is a function, if the parameter in the function is true, it returns 1, otherwise it returns 0;

Calculate the minimum cost value when it is a certain parallax value for each pixel of the image, and accumulate the cost value in 8 directions. After the accumulation, the parallax value with the lowest value is equivalent to the final parallax value of the pixel. Each pixel is calculated one by one to obtain the entire disparity map.
The processing method according to claim 12, wherein the conversion between the disparity value and the depth value can adopt the following formula:

Among them, d p represents the depth value of a certain pixel;

f is the normalized focal length;

b is the baseline distance between the two secondary images;

D p represents the disparity value of the current pixel.
The processing method according to claim 10, wherein the formula used in the edge-directed interpolation is as follows:

Among them, m and n are low-resolution and high-resolution image grids before and after interpolation;

y[n] represents the depth map generated after interpolation;

x[m] and
Represents the original depth map and the corrected map;

Represents the reference data of the second image 130;

S and R respectively represent the data model of the second image 130 and the operator of the edge-directed rendering step;

λ is the gain of the correction process;

k is the iteration index.