CN113438388A

CN113438388A - Processing method, camera assembly, electronic device, processing device and medium

Info

Publication number: CN113438388A
Application number: CN202110762101.2A
Authority: CN
Inventors: 胡攀
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-07-06
Filing date: 2021-07-06
Publication date: 2021-09-24

Abstract

The application discloses a processing method, a camera assembly, an electronic device, a processing device and a medium. The processing method is used for the camera assembly. The camera assembly includes a lens, an image sensor, and a driver. The image sensor can acquire external light through the lens and generate a corresponding original image. The driving member can drive the image sensor to move between a first position and a second position. The processing method comprises the following steps: acquiring a first original image of a target scene generated by an image sensor at a first position; acquiring a second original image of the target scene generated by the image sensor at a second position; and determining the depth image of the target scene according to the first original image, the second original image, the first position, the second position and the focal length of the lens. The processing method, the camera assembly, the electronic equipment, the processing device and the medium have low requirements on software and hardware, and the accuracy of the depth value in the obtained depth image is high.

Description

Processing method, camera assembly, electronic device, processing device and medium

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to a processing method, a camera assembly, an electronic device, a processing apparatus, and a medium.

Background

In the related art, regarding how to acquire a depth map in an electronic device, technical solutions generally adopted are bi-shot image-based, structured light-based, TOF-based, DFD-based, and AI-single-shot-based. In the scheme of obtaining the depth map based on double-shooting, structured light and TOF, except a module which needs the shooting and photographing function, a black-and-white module, a structured light module or a TOF module and the like are required to be added specially for the depth map calculation, so that the hardware cost and the power consumption overhead are high; in the scheme of obtaining the depth map based on the DFD and AI single shot, the requirement on the algorithm is high, and the precision of the obtained depth is low. Therefore, a method for acquiring a depth map with lower requirements on software and hardware and higher accuracy is needed.

Disclosure of Invention

Embodiments of the present application provide a processing method, a camera assembly, an electronic device, a processing apparatus, and a medium.

The processing method of the embodiment of the application is used for the camera assembly. The camera assembly includes a lens, an image sensor, and a driving member. The image sensor can acquire external light through the lens and generate a corresponding original image. The driving member can drive the image sensor to move between a first position and a second position. The processing method comprises the following steps: acquiring a first original image of a target scene generated by the image sensor at the first position; acquiring a second original image of the target scene generated by the image sensor at the second position; and determining a depth image of the target scene according to the first original image, the second original image, the first position, the second position and the focal length of the lens.

The camera assembly of the embodiment of the application comprises a lens, an image sensor, a driving piece and a processor. The image sensor can acquire external light through the lens and generate a corresponding original image. The driving member can drive the image sensor to move between a first position and a second position. The processor is configured to: acquiring a first original image of a target scene generated by the image sensor at the first position; acquiring a second original image of the target scene generated by the image sensor at the second position; and determining a depth image of the target scene according to the first original image, the second original image, the first position, the second position and the focal length of the lens.

The electronic equipment of this application embodiment includes casing and above-mentioned embodiment the camera subassembly, the camera subassembly with the casing combines.

The processing device of the embodiment of the application is used for the camera assembly. The camera assembly includes a lens, an image sensor, and a driving member. The image sensor can acquire external light through the lens and generate a corresponding original image. The driving member can drive the image sensor to move between a first position and a second position. The processing device comprises a first obtaining module, a second module and a determining module. The first acquisition module is used for acquiring a first original image of a target scene generated at the first position by the image sensor. A second module is configured to obtain a second raw image of the target scene generated by the image sensor at the second location. The determining module is configured to determine a depth image of the target scene according to the first original image, the second original image, the first position, the second position, and the focal length of the lens.

The computer-readable storage medium of the present embodiment stores thereon a computer program, which is characterized by implementing the steps of the processing method described in the above embodiment when the program is executed by a processor.

In the processing method, the camera assembly, the electronic device, the processing device and the medium, the image sensor can move under the driving of the driving part, so that the depth image of the target scene is determined according to the first original image of the target scene generated at the first position by the image sensor, the second original image of the target scene generated at the second position by the image sensor, the first position, the second position and the focal length of the lens, the requirements on software and hardware are low, and the accuracy of the depth value in the obtained depth image is high.

Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.

Drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a schematic flow diagram of a treatment process according to an embodiment of the present application;

FIG. 2 is a schematic view of a camera assembly according to an embodiment of the present application;

FIG. 3 is a schematic view of a processing apparatus according to an embodiment of the present application;

FIG. 4 is a schematic view of a camera assembly according to an embodiment of the present application;

FIG. 5 is a schematic diagram of an exposure timing of a processing method according to an embodiment of the present application;

FIG. 6 is a schematic diagram of the overall control sequence of the processing method of the embodiment of the present application;

FIG. 7 is a schematic flow chart of a treatment method according to an embodiment of the present application;

FIG. 8 is a schematic view of a processing apparatus according to an embodiment of the present application;

FIG. 9 is a schematic view of a processing method according to an embodiment of the present application;

FIG. 10 is a schematic diagram of the processing method of an embodiment of the present application;

FIG. 11 is a schematic flow chart of a treatment method according to an embodiment of the present application;

FIG. 12 is a schematic view of a processing apparatus according to an embodiment of the present application;

FIG. 13 is a schematic flow chart of a treatment method according to an embodiment of the present application;

FIG. 14 is a schematic view of a processing apparatus according to an embodiment of the present application;

FIG. 15 is a schematic flow chart of a treatment method according to an embodiment of the present application;

FIG. 16 is a schematic flow chart of a treatment method according to an embodiment of the present application;

FIG. 17 is a schematic view of a processing apparatus according to an embodiment of the present application;

FIG. 18 is a schematic flow chart of a treatment method according to an embodiment of the present application;

FIG. 19 is a schematic view of a processing apparatus according to an embodiment of the present application;

FIG. 20 is a schematic flow chart of a treatment method according to an embodiment of the present application;

FIG. 21 is a schematic diagram of the treatment process of an embodiment of the present application;

FIG. 22 is a schematic flow chart of a treatment method according to an embodiment of the present application;

FIG. 23 is a schematic view of a processing apparatus according to an embodiment of the present application;

fig. 24 is a schematic diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative and are only for the purpose of explaining the present application and are not to be construed as limiting the present application.

In the description of the embodiments of the present application, the terms "first", "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", may explicitly or implicitly include one or more of the described features. In the description of the embodiments of the present application, "a plurality" means two or more unless specifically defined otherwise.

Referring to fig. 1-3, the processing method of the present application is applied to a camera assembly 10. The camera assembly 10 includes a Lens 12(Lens), an image sensor 14, and a driver 16. The image sensor 14 is capable of capturing ambient light through the lens 12 and generating a corresponding raw image. The driver 16 is capable of driving the image sensor 14 between a first position and a second position. The processing method comprises the following steps:

01: acquiring a first raw image of the target scene generated by the image sensor 14 at a first location;

03: acquiring a second raw image of the target scene generated by the image sensor 14 at a second location;

05: and determining the depth image of the target scene according to the first original image, the second original image, the first position, the second position and the focal length of the lens 12.

The processing method according to the embodiment of the present application can be realized by the camera head assembly 10 according to the embodiment of the present application. Specifically, the camera assembly 10 includes a lens 12, an image sensor 14, a driving member 16 and a processor 18, wherein the image sensor 14 can acquire external light through the lens 12 and generate a corresponding original image, the driving member 16 can drive the image sensor 14 to move between a first position and a second position, and the processor 18 is configured to: acquiring a first raw image of the target scene generated by the image sensor 14 at a first location; acquiring a second raw image of the target scene generated by the image sensor 14 at a second location; and determining the depth image of the target scene according to the first original image, the second original image, the first position, the second position and the focal length of the lens 12.

The processing method according to the embodiment of the present application can be realized by the processing apparatus 200 according to the embodiment of the present application. Specifically, the processing device 200 is for the camera head assembly 10. The camera assembly 10 includes a lens 12, an image sensor 14, and a driver 16. The image sensor 14 is capable of capturing ambient light through the lens 12 and generating a corresponding raw image. The driver 16 is capable of driving the image sensor 14 between a first position and a second position. The processing device 200 comprises a first obtaining module 21, a second obtaining module 23 and a first determining module 25. The first acquiring module 21 is configured to acquire a first raw image of a target scene generated by the image sensor 14 at a first position. The second acquiring module 23 is configured to acquire a second original image of the target scene generated by the image sensor 14 at a second position. The first determining module 25 is configured to determine a depth image of the target scene according to the first original image, the second original image, the first position, the second position, and the focal length of the lens 12.

The image sensor 14 of the present application can move under the driving of the driving element 16, so as to determine the depth image of the target scene according to the first original image of the target scene generated by the image sensor 14 at the first position, the second original image of the target scene generated by the image sensor 14 at the second position, the first position, the second position and the focal length of the lens 12.

It can be understood that, in the related art, how to acquire a depth map in an electronic device can be mainly divided into two main types of implementation manners. One type is to rely on a multi-mode group (double shooting) or a special module (structured light/TOF) to obtain a depth image, the implementation mode needs to analyze and solve the scene depth through extra information such as multi-view angle or special imaging provided by the multi-mode group, the hardware cost of the implementation mode is high, extra modules are required to be added specially aiming at the requirement of depth image calculation besides the module required by the normal shooting and photographing function, such as a black-and-white module for double shooting, a TOF module, a structured light module and the like, besides the cost brought by the added modules, the implementation mode influences the size, the weight, the multi-shooting arrangement space and the like of the electronic equipment, and the multi-mode group causes a large amount of power consumption overhead, so that the temperature rise and the endurance of the electronic equipment are also influenced. The other type of the method is that the method depends on the fitting of redundant data and prior information, based on a set model fitting optimal solution, a data set of a matching relation between a scene picture and a depth map or a data set of a relation between an out-of-focus image and a focusing distance is established through depth learning, iterative adjustment is carried out to obtain a parameter with the minimum difference from GT under the set model constraint, the depth information calculation can be carried out on the out-of-focus picture or the scene picture by combining the parameter and the model, and the calculation precision of the implementation mode is limited. For example, based on a DFD method, multiple frames of images with different focusing distances need to be acquired, if reasonable accuracy is achieved, hundreds or even thousands of frames of images may be required, and it is ensured that a camera is still and a target scene is completely unchanged, then the sharpest focusing position is indexed for each point in the image, and the object distance of each point is obtained based on a focusing formula. The method based on AI single shot estimation has a huge demand on a data set, because different cameras have different focal lengths, distortion and imaging modes, and the relative relationship between a 2D image plane and a 3D space is inconsistent, each camera needs to reconstruct the data set, and considering a complex 3D measurement device, alignment with a 2D image, and the like, the construction of data is a complex process, and errors are easily generated only by relying on the data set and prior constraints due to the uncontrollable nature of a deep learning process and information loss from a single shot 2D image to the 3D space.

In the processing method of the embodiment of the application, the camera assembly 10 shoots two frames of images of the image sensor 14 at different positions in a sensor shift mode based on an optical anti-shake technology of sensor shift, and generates a depth image of a target scene based on parallax of the two frames of images, so that the depth image can be generated through the sensor shift module, the hardware cost is low, the power consumption overhead is low, and the accuracy of the depth value in the obtained depth image is also high. In addition, camera subassembly 10 is relevant with sensor shift's optical anti-shake technique, through combining with this technique, can multiplex module development production, realizes technical iteration, promotes industry development.

Specifically, the camera assembly 10 may have both an optical anti-shake function and a depth image generation function. The movement of the image sensor 14 driven by the driving member 16 may include displacement (sensor shift) or rotation (sensor tilt), and the beneficial effects and embodiments of the processing method of the present application will be described below by taking the sensor shift as an example. The drive member 16 may comprise a Voice Coil Motor (VCM). In some embodiments, the driver 16 can drive the image sensor 14 to move within a predetermined plane, for example, the driver 16 can drive the image sensor 14 to displace to the left or to displace to the right within the predetermined plane. In general, the center of the lens 12 is collinear with the center of the image sensor 14, and when the driver 16 drives the image sensor 14 to move, the position of the center of the lens 12 changes, the center of the lens 12 is not collinear with the center of the image sensor 14, and the same object point generates different imaging points on the image sensor 14. The first position and the second position may be two different positions on the preset plane. In some embodiments, the image sensor 14 includes a plurality of photosensitive units, all of which are arranged in the same plane, and in order to quickly determine the depth image of the target scene, a plane parallel to the plane of the photosensitive units may be used as the preset plane, that is, the driving member 16 drives the image sensor 14 to move in the preset plane parallel to the plane of the photosensitive units. For example, the driver 16 can drive the image sensor 14 to laterally displace or longitudinally displace within a predetermined plane parallel to the plane of the light sensing unit.

Further, the target scene may be understood as a scene for which depth values need to be calculated. The first original image and the second original image each include a target scene, and the first original image and the second original image may further include a non-target scene for which depth values do not need to be calculated. When it is required to determine the depth information of the target scene, the driving element 16 drives the image sensor 14 to a first position, the image sensor 14 generates a first original image including the target scene according to the external light received at the first position, then the driving element 16 drives the image sensor 14 to a second position, the image sensor 14 generates a second original image including the target scene according to the external light received at the second position, and the processor 18 determines the depth image of the target scene according to the first original image, the second original image, the first position, the second position and the focal length of the lens 12. One of the first raw image and the second raw image and the depth image may be an output image of the image sensor 14.

It can be understood that, in order to avoid that the first original image and the second original image are blurred themselves and the difference between the first original image and the second original image is large (electronic device movement or target scene movement) due to the excessively long exposure time and the excessively long inter-frame interval time, the first original image and the second original image can be acquired through exposure in rapid succession.

In certain embodiments, steps 01, 03 and 05 may also be implemented by the image sensor 14. Therefore, on one hand, accurate time sequence control between the motion of the image sensor 14 and the motion of the image sensor 14 can be realized, and the image sensor 14 is ensured to be moved in place and kept still during exposure; on the other hand, the first position and the second position can be directly determined, so that the distance between the first position and the second position is determined; in addition, the image sensor 14 has a function of directly outputting the depth image, and the normal image and the corresponding depth image can be directly output by packaging by the image sensor 14, so that design and development convenience is provided for a back-end application to use the depth image frame by frame. Specifically, referring to fig. 4, the Image sensor 14 may include a Double Data Rate (DDR), an arithmetic module (Image processing & depth computing), and a Mobile Industry Processor Interface (MIPI). The camera assembly 10 may include an Application Processor (AP). The DDR is able to store two consecutive frames of the first original image and the second original image, thereby ensuring that the entire contents of the first original image and the second original image can be loaded in the DDR before computing the feature. The operation module can perform rapid and low-power-consumption feature matching and depth calculation on the first original image and the second original image, so that the normal image frame rate is not influenced by the depth image function. The encapsulated normal image and depth image can be output through the MIPI and output in the same request or the same interrupt, so that the application processor at the back end can use more convenient software design for data utilization. The output normal image may be the first original image or the second original image.

Further, considering that the image sensor 14 needs to know the first position and the second position and can be accurately synchronized with the frame exposure, an optical anti-shake control logic may also be built into the image sensor 14, so that the motion of the image sensor 14 can be directly synchronized with the frame alignment (Vsync) and line alignment (Hsync) timing of the image sensor 14, and therefore, in the embodiment shown in fig. 4, the image sensor 14 has a direct hardware connection (e.g., SPI) with the driving circuit (Driver IC) for control.

In addition, in some embodiments, the image sensor 14 performs exposure by using a Rolling Shutter (Rolling Shutter), and the exposure timing is as shown in fig. 5, where Vsync is the middle time point of a single frame image exposure, and Hsync is the middle time point of each line exposure, so that the complete control timing of the image sensor 14 can be as shown in fig. 6. In the control timing shown in fig. 6, the Frame Image data with depth data of the Image Sensor 14 is output until the first and second original images are exposed and the calculation of the depth Image is completed, and the control of the position of the Image Sensor 14 is completed before each exposure (OIS Sensor Shift).

Referring to fig. 7 and 8, in some embodiments, the target scene includes target object points, and step 05 includes:

051: determining the parallax of a target object point in a target scene in the first original image and the second original image;

053: determining a first distance between the first location and the second location;

055: from the parallax, the first distance, and the focal length of the lens 12, the depth values of the target object points are determined to form a depth image of the target scene.

The processing method of the above embodiment can be implemented by the camera head assembly 10 of the embodiment of the present application. In particular, the processor 18 is configured to determine a disparity of a target object point in the target scene in the first original image and the second original image, and to determine a first distance between the first location and the second location, and to determine a depth value of the target object point based on the disparity, the first distance, and the focal length of the lens 12 to form a depth image of the target scene.

The processing method according to the above embodiment can be realized by the processing apparatus 200 according to the embodiment of the present application. Specifically, the first determination module 25 includes a first determination unit 2511, a second determination unit 2513, and a third determination unit 2515. The first determining unit 2511 is configured to determine the parallax of the target point in the first original image and the second original image in the target scene. The second determining unit 2513 is configured to determine a first distance between the first position and the second position. The third determining unit 2515 is configured to determine depth values of the target object points to form a depth image of the target scene according to the parallax, the first distance, and the focal length of the lens 12.

Therefore, the depth value of the target object point can be accurately determined based on the binocular vision parallax ranging principle. It will be appreciated that when the driver 16 drives the image sensor 14 to move in a predetermined plane parallel to the plane of the light sensing units, the first and second raw images correspond to images acquired by two camera assemblies 10 located in the same plane, thus satisfying the conditions of the binocular vision parallax ranging principle.

Specifically, since the first original image is obtained when the image sensor 14 is at the first position, the second original image is obtained when the image sensor 14 is at the second position, and the first position and the second position are different positions in the same plane, positions of corresponding pixel points of the target point in the first original image and the second original image are different, and the target point has a parallax in the first original image and the second original image. In some embodiments, the driving element 16 drives the image sensor 14 to laterally displace in a preset plane parallel to the plane of the photosensitive unit, so that the acquired first original image and second original image directly satisfy epipolar constraint, that is, corresponding pixel points of the same target object point in the first original image and second original image are in common, so that a depth image can be directly acquired based on the parallax principle without using complicated bi-camera calibration and performing additional rotation and translation correction on the images to make the first original image and second original image satisfy the epipolar constraint. Because the corresponding pixel points of the target object point in the first original image and the second original image are positioned in the same row and different columns, the difference value of the column numbers (namely x coordinates) of the corresponding pixel points of the target object point in the first original image and the second original image is determined, and then the parallax of the target object point in the first original image and the second original image can be determined. The first distance is the horizontal distance between the first position and the second position. The depth value is the distance of the target object point from the camera assembly 10.

Further, the target scene may include a plurality of target object points, and after determining the parallax of each object point in the first original image and the second original image, respectively, the depth value of each target object point may be determined according to the determined parallax, the first distance, and the focal length of the lens 12, and then a depth image may be formed according to the depth value of each target object point.

In some embodiments, the target scene includes a plurality of object points, and the target object points are all object points in the target scene or are part of object points in the target scene.

In this way, depth values for all or part of the object points in the target scene may be determined.

Specifically, when the target object points are all object points in the target scene, the depth values of all object points in the target scene are determined to form a depth image, and the depth information in the depth image obtained in this way is relatively comprehensive; when the object points are partial object points in the object scene, depth values of the partial object points in the object scene are determined to form a depth image, so that the amount of calculation of depth information in the obtained depth image can be reduced. It can be understood that, when acquiring a depth image, since it is not necessary to use a depth image with full-size resolution (e.g. 4000x3000) in practical use, but only a depth image with small size (e.g. 640x480) is required, it may be considered that only a small map is used for calculation when acquiring a depth image, or a depth map is acquired by spacing N rows and M columns in the row and column direction, so that the amount of calculation is greatly reduced, the performance of outputting a depth image by the image sensor 14 is improved, and the area and the overhead of a built-in driving circuit of the image sensor 14 are reduced.

Further, in the manner in which the depth values are calculated using the small map to form the depth image, the sizes of the first original image and the second original image may be compressed to obtain the first original image and the second original image of smaller sizes, thereby reducing the amount of calculation required to acquire the depth image. In a manner of obtaining the depth map by N rows and M columns in the row-column direction, all object points of the target scene may be screened according to a preset rule, for example, object points of a specific row and a specific column are selected as target object points in the row-column direction, and then depth values of the object points of the specific row and the specific column are calculated and a depth image is formed, thereby greatly reducing the calculation amount required for obtaining the depth image.

In some embodiments, the depth values of the target object points are: and Z is f × b/d, where f is the focal length of the lens 12, b is the first distance, and d is the parallax of the target point in the first and second original images.

In this way, the depth value of the target object point can be determined from the parallax, the first distance, and the focal length of the lens 12.

Specifically, please refer to FIGS. 9 and 9Fig. 10, in fig. 9, for the target point P, based on the geometrical optics principle, when the image sensor 14 is displaced from the initial position to the left by a distance of m1, the first position is reached, the generated image point is P1, when the image sensor 14 is displaced from the initial position to the right by a distance of m2, the generated image point is P2, the distance between P1 and P2 on the image plane is parallax d, and since the image sensor 14 only generates lateral movement, P1 and P2 are located in the same row and different columns in the image. Further, based on the pinhole imaging model, the geometric imaging process of fig. 9 can be simplified as shown in fig. 10. In the embodiment shown in fig. 8, X2+ X4 ═ m2, X1+ X3 ═ m1, and the image plane distance between the pixels p1 and p2, that is, the horizontal parallax d (different column numbers, that is, X coordinates) is | XR-XL |, which can be obtained from a similar triangle:

since b is m1+ m2, the depth value of the target point P is:

referring to fig. 11 and 12, in some embodiments, before step 03, the processing method further includes:

02: acquiring a middle original image of the target scene generated by the image sensor 14 at a third position, wherein the exposure duration of the middle original image is longer than that of the first original image, and the exposure duration of the middle original image is longer than that of the second original image;

after step 05, the processing method further comprises:

07: and determining a target image according to the intermediate original image and the depth image.

The processing method of the above embodiment can be implemented by the camera head assembly 10 of the embodiment of the present application. Specifically, the processor 18 is configured to obtain an intermediate raw image of the target scene generated by the image sensor 14 at a third position, the third position being located between the first position and the second position, an exposure duration of the intermediate raw image being greater than an exposure duration of the first raw image, the exposure duration of the intermediate raw image being greater than an exposure duration of the second raw image, and to determine the target image from the intermediate raw image and the depth image.

The processing method according to the above embodiment can be realized by the processing apparatus 200 according to the embodiment of the present application. In particular, the processing device 200 further comprises a third acquisition module 27 and a second determination module 29. The third obtaining module 27 is configured to obtain an intermediate original image of the target scene generated by the image sensor 14 at the third position, where an exposure duration of the intermediate original image is greater than an exposure duration of the first original image, and an exposure duration of the intermediate original image is greater than an exposure duration of the second original image. The second determination module 29 is configured to determine the target image according to the intermediate original image and the depth image.

Thus, the intermediate original image and the target image with high definition can be obtained, and the requirements on the image quality of the output image under different states can be met. It will be appreciated that in some embodiments, the first original image and the second original image are obtained in rapid succession, in a double short exposure mode, i.e., the exposure time to obtain the first original image and the exposure time to obtain the second original image are shorter, this may result in poor definition of the first original image and the second original image, and the present embodiment adopts a short long short three exposure mode, that is, acquiring an intermediate original image with a longer exposure time period after acquiring the first original image by short exposure and before acquiring the second original image by short exposure, the depth image calculation is performed by using the first original image and the second original image subjected to head-to-tail short exposure, and the intermediate original image subjected to intermediate long exposure or normal exposure is output as a accompanying normal image, so that the image quality of a target image formed in back-end processing can be ensured.

Specifically, the third position may be the first position or the second position, and the third position may be any position other than the first position and the second position. The intermediate raw image and the depth image may be output images of the image sensor 14. In some embodiments, the intermediate raw image and depth image output by the image sensor 14 are post-processed to obtain a target image.

In some embodiments, the image sensor 14 includes a Mobile Industry Processor Interface (MIPI) capable of transmitting the depth image and the intermediate raw image to an Application Processor (AP) capable of generating the target image from the received depth image and intermediate raw image.

In this way, the image sensor 14 has a function of straightening out the depth image, and the depth image can be directly used by the application processor at the back end.

Referring to fig. 13 and 14, in some embodiments, the first original image includes a plurality of frames, the second original image includes a plurality of frames, and the number of frames of the first original image is the same as that of the second original image, step 05 includes:

057: determining an original depth image of a multi-frame target scene according to the multi-frame first original image, the multi-frame second original image, the first position, the second position and the focal length of the lens 12;

059: and fusing the original depth images of the multi-frame target scene to obtain the depth image of the target scene.

The processing method of the above embodiment can be implemented by the camera head assembly 10 of the embodiment of the present application. Specifically, the processor 18 is configured to determine an original depth image of the multi-frame target scene according to the multi-frame first original image, the multi-frame second original image, the first position, the second position, and the focal length of the lens 12, and to fuse the original depth images of the multi-frame target scene to obtain a depth image of the target scene.

The processing method according to the above embodiment can be realized by the processing apparatus 200 according to the embodiment of the present application. Specifically, the first determination module 25 includes a fourth determination unit 2517 and a first fusion unit 2519. The fourth determining unit 2517 is configured to determine an original depth image of the multi-frame target scene according to the multi-frame first original image, the multi-frame second original image, the first position, the second position and the focal length of the lens 12. The first fusing unit 2519 is configured to fuse the original depth images of the multiple frames of target scenes to obtain depth images of the target scenes.

Therefore, the depth value in the depth image obtained by fusing the multi-frame original depth images is more accurate, and the precision is improved.

Specifically, the number of frames of the original depth image is also the same as the number of frames of the first original image. When a first original image and a second original image are obtained, multiple frames or continuous frames of the first original image and the second original image are shot, multiple frames or continuous frames of the first original image and the second original image can form multiple pairs of the first original image and the second original image, further, original depth images of a target scene corresponding to each pair of the first original image and the second original image can be determined, all the obtained original depth images are fused, and finally, the depth images serving as output can be obtained.

Referring to fig. 15, in some embodiments, the lens 12 includes a first lens and a second lens. The image sensor 14 includes a first image sensor and a second image sensor. The driver 16 includes a first driver and a second driver. The first image sensor can acquire external light through the first lens and generate a corresponding original image. The first driving member can drive the first image sensor to move between a first position and a second position. The second image sensor can acquire external light through the second lens and generate a corresponding original image. The second driving member can drive the second image sensor to move between a first position and a second position. The step 01 comprises the following steps:

012: acquiring a first original image of a target scene generated by a first image sensor at a first position;

step 03 comprises:

032: acquiring a second original image of the target scene generated by a second image sensor at a second position;

step 05 comprises:

052: and determining the depth image of the target scene according to the first original image, the second original image, the first position of the first image sensor, the second position of the second image sensor, the first focal length of the first lens and the second focal length of the second lens.

The processing method of the above embodiment can be implemented by the camera head assembly 10 of the embodiment of the present application. Specifically, the processor 18 is configured to acquire a first raw image of the target scene generated by the first image sensor at a first position, acquire a second raw image of the target scene generated by the second image sensor at a second position, and determine a depth image of the target scene based on the first raw image, the second raw image, the first position of the first image sensor, the second position of the second image sensor, the first focal length of the first lens, and the second focal length of the second lens.

The processing method according to the above embodiment can be realized by the processing apparatus 200 according to the embodiment of the present application. Specifically, the first acquiring module 21 is configured to acquire a first raw image of a target scene generated at a first position by a first image sensor. The second acquiring module 23 is configured to acquire a second original image of the target scene generated by the second image sensor at the second position. The first determining module 25 is configured to determine a depth image of the target scene according to the first original image, the second original image, the first position of the first image sensor, the second position of the second image sensor, the first focal length of the first lens, and the second focal length of the second lens.

Thus, when multiple modules are adopted, the depth value of the target scene in a deeper position can be determined, and the depth value range of the depth image is enlarged. It can be understood that when ranging is performed based on the binocular vision parallax ranging principle, the greater the distance between the two shots, the deeper the depth that can be measured.

Specifically, when multiple modules are employed, the multiple modules are disposed on the same plane. For example, the first lens and the second lens are located on the same plane; the first image sensor and the second image sensor are located on the same plane. A multi-module is understood to mean that the number of image sensors 14 comprises at least two. Each image sensor 14 in the multi-module is movable between a first position and a second position. When multiple modules are used to determine the depth image of the target scene, the depth image of the target scene may be determined according to the original images acquired by different image sensors 14 of different modules, for example, the depth image of the target scene may be determined according to a first original image of the target scene generated by a first image sensor at a first position and a second original image of the target scene generated by a second image sensor at a second position. It can be understood that when the first original image and the second original image are obtained according to different positions of the multi-module different image sensor 14, compared to obtaining the first original image and the second original image according to different positions of the single-module same image sensor 14, the distance between different positions of the different image sensors 14 is greater than the distance between different positions of the same image sensor 14, so that the depth that the multi-module can measure is deeper.

Referring to fig. 16 and 17, in some embodiments, the target scene includes target object points, and step 052 includes:

0521: determining the parallax of a target object point in a target scene in the first original image and the second original image;

0523: determining a second distance between the first position of the first image sensor and the second position of the second image sensor;

0525: determining depth values of the target object points to form a depth image of the target scene according to the parallax, the second distance, the first focal length of the first lens, and the second focal length of the second lens.

The processing method of the above embodiment can be implemented by the camera head assembly 10 of the embodiment of the present application. Specifically, the processor 18 is configured to determine a disparity of a target point in the target scene in the first original image and the second original image, and to determine a second distance between the first position of the first image sensor and the second position of the second image sensor, and to determine a depth value of the target point based on the disparity, the second distance, the first focal length of the first lens, and the second focal length of the second lens to form a depth image of the target scene.

The processing method according to the above embodiment can be realized by the processing apparatus 200 according to the embodiment of the present application. Specifically, the first determination module 25 includes a fifth determination unit 2521, a sixth determination unit 2523, and a seventh determination unit 2525. The fifth determining unit 2521 is configured to determine the parallax of the target point in the target scene in the first original image and the second original image. The sixth determining unit 2523 is configured to determine a second distance between the first position of the first image sensor and the second position of the second image sensor. The seventh determining unit 2525 is configured to determine depth values of the target object points to form a depth image of the target scene according to the parallax, the second distance, the first focal length of the first lens, and the second focal length of the second lens.

Specifically, since the first original image is obtained when the first image sensor is at the first position, the second original image is obtained when the second image sensor is at the second position, and the first position and the second position are different positions in the same plane, positions of corresponding pixel points of the target point in the first original image and the second original image are different, and the target point has a parallax in the first original image and the second original image. In some embodiments, the first driving element drives the first image sensor to laterally displace in a preset plane parallel to a plane in which the photosensitive unit is located, and the second driving element drives the second image sensor to laterally displace in a preset plane parallel to a plane in which the photosensitive unit is located, so that corresponding pixel points of the target object point in the first original image and the second original image are located in the same row and different columns, and a difference between column numbers (i.e., x coordinates) of the corresponding pixel points of the target object point in the first original image and the second original image is determined, so that the parallax of the target object point in the first original image and the second original image can be determined. The second distance is the horizontal distance between the first position of the first image sensor and the second position of the second image sensor. Typically, the second distance is greater than the first distance. The depth value is the distance of the target object point from the camera assembly 10.

Further, the target scene may include a plurality of target object points, and after determining the parallax of each object point in the first original image and the second original image, respectively, the depth value of each target object point may be determined according to the determined parallax, the second distance, and the focal length of the lens 12, and then a depth image may be formed according to the depth value of each target object point.

In some embodiments, the first focal length is the same as the second focal length, and the depth values of the target object points are: and Z is f b/d, wherein f is the first focal length or the second focal length, b is the second distance, and d is the parallax of the target point in the first original image and the second original image.

Thus, the depth value of the target object point can be determined according to the parallax, the second distance, the first focal length or the second focal length.

Referring to fig. 18 and 19, in some embodiments, step 051 or step 0521 includes:

0511: performing feature matching on the first original image and the second original image to determine a first image position of the target point in the first original image and a second image position of the target point in the second original image;

0513: and determining the parallax of the target object point in the target scene in the first original image and the second original image according to the first image position and the second image position.

The processing method of the above embodiment can be implemented by the camera head assembly 10 of the embodiment of the present application. Specifically, the processor 18 is configured to perform feature matching on the first original image and the second original image to determine a first image position of the target point in the first original image and a second image position of the target point in the second original image, and to determine a parallax of the target point in the target scene in the first original image and the second original image according to the first image position and the second image position.

The processing method according to the above embodiment can be realized by the processing apparatus 200 according to the embodiment of the present application. Specifically, the first determination unit 2511 includes a matching sub-unit 25111 and a determination sub-unit 25113. The matching subunit 25111 is configured to perform feature matching on the first original image and the second original image to determine a first image position of the target point in the first original image and a second image position of the target point in the second original image. The determining subunit 25113 is configured to determine the parallax of the target point in the target scene in the first original image and the second original image according to the first image position and the second image position.

Therefore, the parallax of the target object point in the first original image and the second original image can be determined quickly and accurately. It can be understood that, for the case that the image sensor 14 only moves laterally when moving, only the x coordinate difference exists between the corresponding pixel points of the target point in the first original image and the second original image, so that when the parallax calculation is performed by adopting the feature extraction and matching method, the parallax of the target point in the first original image and the second original image can be rapidly obtained. In addition, on the assumption that the same-row constraint and the inter-frame variation difference are small, a dense pixel matching relation from pixel to pixel can be obtained quickly.

Referring to fig. 20, in some embodiments, step 0511 includes:

05111: determining pixel points to be matched in the first original image according to the target object points, wherein the positions of the pixel points to be matched in the first original image are first image positions;

05113: determining a first characteristic of a pixel point to be matched;

05115: determining a second characteristic of each pixel point in a second original image;

05117: and matching the first characteristics of the pixel points to be matched with the second characteristics of each pixel point in the second original image to determine the pixel points in the second original image matched with the pixel points to be matched, wherein the positions of the pixel points in the second original image matched with the pixel points to be matched in the second original image are the positions of the second image.

The processing method of the above embodiment can be implemented by the camera head assembly 10 of the embodiment of the present application. Specifically, the processor 18 is configured to determine, according to the target object point, a pixel point to be matched in the first original image, where a position of the pixel point to be matched in the first original image is a first image position, and is configured to determine a first feature of the pixel point to be matched, and is configured to determine a second feature of each pixel point in the second original image, and is configured to match the first feature of the pixel point to be matched with the second feature of each pixel point in the second original image, so as to determine a pixel point in the second original image that is matched with the pixel point to be matched, where a position of the pixel point in the second original image that is matched with the pixel point to be matched is a second image position.

The processing method according to the above embodiment can be realized by the processing apparatus 200 according to the embodiment of the present application. Specifically, the matching subunit 25111 is configured to determine, according to the target object point, a to-be-matched pixel point in the first original image, where a position of the to-be-matched pixel point in the first original image is a first image position, and is configured to determine a first feature of the to-be-matched pixel point, and is configured to determine a second feature of each pixel point in the second original image, and is configured to match the first feature of the to-be-matched pixel point with the second feature of each pixel point in the second original image, so as to determine a pixel point in the second original image that is matched with the to-be-matched pixel point, where a position of the pixel point in the second original image that is matched with the to-be-matched pixel point is a second image position.

Thus, the positions of the target point in the first original image and the second original image can be determined quickly and accurately.

Specifically, the pixel point to be matched is the pixel point of the target object point corresponding to the first original image. The number of the pixel points to be matched can be the same as that of the target object points. When the first original image and the second original image are matched, for each pixel point to be matched in the first original image, pixel points in the same row and a certain column range of the second original image are searched for matching, the best matching point is taken as a matching pair, and the column distance between two pixel points in the matching pair is the parallax of the corresponding target object point in the first original image and the second original image.

Further, please refer to fig. 21, in order to eliminate the pixel value difference of the pixel points corresponding to the same object point between frames caused by external changes such as illumination and noise during feature matching, the first feature and the second feature may be characterized by using a more robust neighborhood descriptor, such as a gradient histogram of a neighborhood of the pixel point 3x3, 5x5, or 7x 7. Meanwhile, some change constraints and interpolation fitting can be performed on the image plane to ensure that a smoother and continuous depth image is obtained by combining the continuity of the picture.

It should be noted that, in some embodiments, since the feature matching is performed in the raw domain, the descriptors and matching method used in the foregoing conventional image matching method need to be modified appropriately and add certain constraints to adapt to the RGGB, BGGR, GBGR, and even the Quad bayer and other CFA pattern arrangement forms. For example, in the feature matching, the R pixel is matched with only the R pixel, the G pixel is matched with only the G pixel, and in the descriptor calculation, only the gradient histogram of the R pixel is counted for the descriptor of the R pixel, or the matching is performed using the RGB three-channel histogram.

Referring to fig. 22 and 23, in some embodiments, the first original image includes a plurality of frames, the second original image includes a plurality of frames, and the number of frames of the first original image is the same as that of the second original image, step 052 includes:

0527: determining an original depth image of a multi-frame target scene according to a plurality of frames of first original images, a plurality of frames of second original images, a first position of a first image sensor, a second position of a second image sensor, a first focal length of a first lens and a second focal length of a second lens;

0529: and fusing the original depth images of the multi-frame target scene to obtain the depth image of the target scene.

The processing method of the above embodiment can be implemented by the camera head assembly 10 of the embodiment of the present application. Specifically, the processor 18 is configured to determine an original depth image of the multi-frame target scene according to the multi-frame first original image, the multi-frame second original image, the first position of the first image sensor, the second position of the second image sensor, the first focal length of the first lens, and the second focal length of the second lens, and to fuse the original depth images of the multi-frame target scene to obtain a depth image of the target scene.

The processing method according to the above embodiment can be realized by the processing apparatus 200 according to the embodiment of the present application. Specifically, the first determination module 25 includes an eighth determination unit 2527 and a second fusion unit 2529. The eighth determining unit 2527 is configured to determine an original depth image of the multi-frame target scene from the plurality of frames of the first original image, the plurality of frames of the second original image, the first position of the first image sensor, the second position of the second image sensor, the first focal length of the first lens, and the second focal length of the second lens. The second fusion unit 2529 is configured to fuse the original depth images of the multiple frames of the target scene to obtain a depth image of the target scene.

In some embodiments, the image sensor 14 includes a double rate synchronous dynamic random access memory capable of storing the first original image and the second original image and an arithmetic module capable of determining a depth image of the target scene based on the first original image, the second original image, the first location, the second location, and the focal length of the lens 12.

Thus, the image sensor 14 can directly output the depth image, and is convenient for the back-end AP to directly use.

In particular, the operation module can be used to match the same target point in the first original image and the second original image, and can also be used to calculate the depth value of the target point.

Referring to fig. 24, an electronic device 100 according to an embodiment of the present disclosure includes a housing 30 and the camera assembly 10 according to the above embodiment, and the camera assembly 10 is combined with the housing 30.

It should be noted that the above explanation of the processing method and the embodiment and the advantageous effects of the camera assembly 10 are also applicable to the electronic apparatus 100 of the present embodiment, and are not detailed herein to avoid redundancy.

Specifically, in the embodiment of fig. 24, the electronic device 100 is a mobile phone, and in other embodiments, the electronic device 100 may also be a tablet computer, a notebook computer, an intelligent wearable device (a smart watch, a smart bracelet, or the like), a head display device, a game machine, a smart home appliance, a display, or any other device having an image capturing function. The camera head assembly 10 is coupled to the housing 30, for example, the camera head assembly 10 may be mounted within the housing 30.

The computer-readable storage medium of the embodiments of the present application stores thereon a computer program, which, when executed by a processor, implements the steps of the processing method of any of the embodiments described above.

For example, in the case where the program is executed by a processor, the steps of the following processing method are implemented:

It will be appreciated that the computer program comprises computer program code. The computer program code may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), software distribution medium, and the like. The Processor may be a central processing unit, or may be other general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, or the like.

In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.

Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.

Claims

1. A processing method for a camera assembly, the camera assembly including a lens, an image sensor and a driving member, the image sensor being capable of capturing ambient light through the lens and generating a corresponding original image, the driving member being capable of driving the image sensor to move between a first position and a second position, the processing method comprising:

acquiring a first original image of a target scene generated by the image sensor at the first position;

acquiring a second original image of the target scene generated by the image sensor at the second position;

and determining a depth image of the target scene according to the first original image, the second original image, the first position, the second position and the focal length of the lens.

2. The processing method of claim 1, wherein the target scene comprises a target object point, and wherein determining the depth image of the target scene from the first original image, the second original image, the first location, the second location, and the focal length of the lens comprises:

determining the parallax of the target object point in the target scene in the first original image and the second original image;

determining a first distance between the first location and the second location;

determining depth values for the target object points to form a depth image of the target scene based on the disparity, the first distance, and a focal length of the lens.

3. The processing method according to claim 2, wherein the target scene includes a plurality of object points, and the target object points are all of the object points in the target scene or are some of the object points in the target scene.

4. The processing method according to claim 2, wherein the depth values of the target object points are: and Z is f b/d, wherein f is the focal length of the lens, b is the first distance, and d is the parallax of the target point in the first original image and the second original image.

5. The processing method of claim 1, wherein prior to said acquiring a second raw image of the target scene generated by the image sensor at the second location, the processing method further comprises:

acquiring a middle original image of the target scene generated by the image sensor at a third position, wherein the exposure duration of the middle original image is longer than that of the first original image, and the exposure duration of the middle original image is longer than that of the second original image;

after the determining the depth image of the target scene according to the first original image, the second original image, the first position, the second position, and the focal length of the lens, the processing method further includes:

and determining a target image according to the intermediate original image and the depth image.

6. The process of claim 5, wherein the image sensor comprises a mobile industry processor interface, the camera assembly comprises an application processor, the mobile industry processor interface is capable of transmitting the depth image and the intermediate raw image to the application processor, and the application processor is capable of generating the target image from the received depth image and intermediate raw image.

7. The processing method of claim 1, wherein the first original image comprises a plurality of frames, the second original image comprises a plurality of frames, the number of frames of the first original image is the same as the number of frames of the second original image, and the determining the depth image of the target scene according to the first original image, the second original image, the first position, the second position and the focal length of the lens comprises:

determining a plurality of frames of original depth images of the target scene according to the plurality of frames of the first original images, the plurality of frames of the second original images, the first position, the second position and the focal length of the lens;

and fusing multiple frames of original depth images of the target scene to obtain a depth image of the target scene.

8. The processing method according to claim 1, wherein the lens comprises a first lens and a second lens, the image sensor comprises a first image sensor and a second image sensor, the driving member comprises a first driving member and a second driving member, the first image sensor can acquire the ambient light through the first lens and generate a corresponding original image, the first driving member can drive the first image sensor to move between a first position and a second position, the second image sensor can acquire the ambient light through the second lens and generate a corresponding original image, and the second driving member can drive the second image sensor to move between the first position and the second position;

the acquiring a first raw image of a target scene generated by the image sensor at the first location comprises:

acquiring the first original image of the target scene generated by the first image sensor at the first position;

the acquiring a second raw image of the target scene generated by the image sensor at the second location comprises:

acquiring the second original image of the target scene generated by the second image sensor at the second position;

determining a depth image of the target scene according to the first original image, the second original image, the first position, the second position, and the focal length of the lens, including:

and determining a depth image of the target scene according to the first original image, the second original image, the first position of the first image sensor, the second position of the second image sensor, the first focal length of the first lens and the second focal length of the second lens.

9. The processing method of claim 8, wherein the target scene comprises a target object point, and wherein determining the depth image of the target scene from the first raw image, the second raw image, the first position of the first image sensor, the second position of the second image sensor, the first focal length of the first lens, and the second focal length of the second lens comprises:

determining a second distance between a first location of the first image sensor and a second location of the second image sensor;

determining depth values of the target object points to form a depth image of the target scene according to the parallax, the second distance, the first focal length of the first lens, and the second focal length of the second lens.

10. The processing method of claim 9, wherein the target scene comprises a plurality of object points, and the target object points are all of the object points in the target scene or are part of the object points in the target scene.

11. The processing method according to claim 9, wherein the first focal distance is the same as the second focal distance, and the depth value of the target point is: and Z is f b/d, wherein f is the first focal length or the second focal length, b is the second distance, and d is the parallax of the target point in the first original image and the second original image.

12. The processing method according to claim 2 or 9, wherein the determining the disparity of the target object point in the target scene in the first original image and the second original image comprises:

performing feature matching on the first original image and the second original image to determine a first image position of the target point in the first original image and a second image position of the target point in the second original image;

and determining the parallax of the target object point in the target scene in the first original image and the second original image according to the first image position and the second image position.

13. The processing method according to claim 12, wherein said performing feature matching on the first original image and the second original image to determine a first image position of the target point in the first original image and a second image position of the target point in the second original image comprises:

determining a pixel point to be matched in the first original image according to the target object point, wherein the position of the pixel point to be matched in the first original image is the position of the first image;

determining a first characteristic of the pixel point to be matched;

determining a second characteristic of each pixel point in the second original image;

and matching the first characteristics of the pixel points to be matched with the second characteristics of each pixel point in the second original image to determine the pixel points matched with the pixel points to be matched in the second original image, wherein the positions of the pixel points matched with the pixel points to be matched in the second original image are the positions of the second image.

14. The processing method of claim 8, wherein the first original image comprises a plurality of frames, wherein the second original image comprises a plurality of frames, wherein the number of frames of the first original image is the same as the number of frames of the second original image, and wherein determining the depth image of the target scene from the first original image, the second original image, the first position of the first image sensor, the second position of the second image sensor, the first focal length of the first lens, and the second focal length of the second lens comprises:

determining a plurality of frames of original depth images of the target scene according to a plurality of frames of the first original image, a plurality of frames of the second original image, a first position of the first image sensor, a second position of the second image sensor, a first focal length of the first lens and a second focal length of the second lens;

15. The processing method of claim 1, wherein the image sensor comprises a double rate synchronous dynamic random access memory capable of storing the first and second raw images and an arithmetic module capable of determining a depth image of the target scene from the first raw image, the second raw image, the first location, the second location, and a focal length of the lens.

16. A camera assembly, comprising a lens, an image sensor, a driving member and a processor, wherein the image sensor is capable of capturing ambient light through the lens and generating a corresponding original image, the driving member is capable of driving the image sensor to move between a first position and a second position, and the processor is configured to:

17. An electronic device, comprising:

a housing; and

the camera assembly of claim 16, in combination with the housing.

18. A processing device for a camera assembly, the camera assembly including a lens, an image sensor and a driving member, the image sensor being capable of capturing ambient light through the lens and generating a corresponding original image, the driving member being capable of driving the image sensor to move between a first position and a second position, the processing device comprising:

a first acquisition module, configured to acquire a first original image of a target scene generated by the image sensor at the first position;

a second module for acquiring a second raw image of the target scene generated by the image sensor at the second location;

a determining module, configured to determine a depth image of the target scene according to the first original image, the second original image, the first position, the second position, and the focal length of the lens.

19. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, carries out the steps of the processing method of any one of claims 1 to 15.