WO2021093635A1

WO2021093635A1 - Image processing method and apparatus, electronic device, and computer readable storage medium

Info

Publication number: WO2021093635A1
Application number: PCT/CN2020/126122
Authority: WO
Inventors: 贾玉虎
Original assignee: Oppo广东移动通信有限公司
Priority date: 2019-11-12
Filing date: 2020-11-03
Publication date: 2021-05-20
Also published as: CN112866549A; CN112866549B

Abstract

An image processing method, comprising: acquiring a preview image; dividing the preview image into at least two sub-regions; acquiring a phase difference corresponding to each sub-region of the at least two sub-regions; determining at least two target phase differences from phase differences corresponding to each sub-region, wherein the at least two target phase differences include a target foreground phase difference and a target background phase difference; focusing according to each target phase difference to obtain an image corresponding to each target phase difference; and combining the images corresponding to each target phase difference to obtain a full-quasi-focus image.

Description

Image processing method and device, electronic equipment, and computer readable storage medium

Cross-references to related applications

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on November 12, 2019, the application number is 201911101432.0, and the invention title is "Image processing methods and devices, electronic equipment, computer-readable storage media", and its entire contents Incorporated in this application by reference.

Technical field

This application relates to the field of image processing technology, and in particular to an image processing method and device, electronic equipment, and computer-readable storage media.

Background technique

The current focusing method is to focus within a rectangular frame. The rectangular frame will include the foreground and background, and the focus can only be focused to a certain position to achieve quasi-focus. When focusing on the foreground, the background is out of focus; when focusing on the background, the foreground is out of focus. Traditional image processing methods have the problem of low image clarity.

Summary of the invention

According to various embodiments of the present application, an image processing method, apparatus, electronic device, and computer-readable storage medium are provided.

An image processing method applied to electronic equipment, including:

Obtain a preview image;

Dividing the preview image into at least two sub-areas;

Acquiring a phase difference corresponding to each of the at least two sub-regions;

Determining at least two target phase differences from the phase differences corresponding to each sub-region, where the at least two target phase differences include a target foreground phase difference and a target background phase difference;

Performing focusing according to the phase difference of each target to obtain an image corresponding to the phase difference of each target;

Synthesis is performed according to the images corresponding to the phase difference of each target to obtain a fully in-focus image.

An image processing device, characterized in that it comprises:

Preview image acquisition module for acquiring preview images;

A dividing module, configured to divide the preview image into at least two sub-areas;

A phase difference acquiring module, configured to acquire the phase difference corresponding to each of the at least two sub-regions;

The phase difference acquisition module is further configured to determine at least two target phase differences from the phase difference corresponding to each sub-region, and the at least two target phase differences include a target foreground phase difference and a target background phase difference;

A focusing module, configured to perform focusing according to the phase difference of each target to obtain an image corresponding to the phase difference of each target;

The synthesis module is used for synthesizing the image corresponding to each target phase difference to obtain a fully in-focus image.

An electronic device includes a memory and a processor, and a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the following steps:

Obtain a preview image;

Dividing the preview image into at least two sub-areas;

A computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

Obtain a preview image;

Dividing the preview image into at least two sub-areas;

The above-mentioned image processing method and device, electronic equipment, computer-readable storage medium, obtaining a preview image, dividing the preview image into at least two sub-areas, obtaining the phase difference corresponding to each of the at least two sub-areas, and corresponding to each sub-areas At least two target phase differences are determined in the phase difference, and the at least two target phase differences include the target foreground phase difference and the target background phase difference. Focus is performed according to each target phase difference to obtain an image corresponding to each target phase difference. Acquire at least two images in different focal points, one of which is the background in-focus image and the other is the foreground in-focus image, and synthesize the images corresponding to the phase difference of each target to obtain a fully in-focus image, which can be out of focus The image with less area improves the sharpness of the image.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

Fig. 1 is an application environment diagram of an image processing method in an embodiment.

Fig. 2 is a flowchart of an image processing method in an embodiment.

Fig. 3 is a schematic diagram of the principle of phase focusing in an embodiment.

FIG. 4 is a schematic diagram of phase detection pixel points arranged in pairs in the pixel points included in the image sensor in an embodiment.

Fig. 5 is a schematic diagram of a part of the structure of an electronic device in an embodiment.

FIG. 6 is a schematic structural diagram of a part of the image sensor 504 in an embodiment.

FIG. 7 is a schematic diagram of the structure of pixels in an embodiment.

FIG. 8 is a schematic diagram of the internal structure of an image sensor in an embodiment.

FIG. 9 is a schematic diagram of the pixel point group Z in an embodiment.

FIG. 10 is a schematic diagram of a process of obtaining the phase difference corresponding to each sub-region in an embodiment.

FIG. 11 is a schematic diagram of performing segmentation processing on the target brightness map in the first direction in an embodiment.

FIG. 12 is a schematic diagram of performing segmentation processing on the target brightness map in the second direction in an embodiment.

Fig. 13 is a schematic flow chart of synthesizing to obtain a full in-focus image in an embodiment.

Fig. 14 is a schematic flow chart of synthesizing to obtain a full in-focus image in another embodiment.

Fig. 15 is a schematic flow chart of synthesizing a full in-focus image in another embodiment.

Fig. 16 is a structural block diagram of an image processing apparatus according to an embodiment.

Fig. 17 is a schematic diagram of the internal structure of an electronic device in an embodiment.

Detailed ways

In order to make the purpose, technical solutions, and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the application, and not used to limit the application.

It can be understood that the terms "first", "second", etc. used in this application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first data from another. For example, without departing from the scope of the present application, the first average phase difference may be referred to as the second average phase difference, and similarly, the second average phase difference may be referred to as the first average phase difference. The first mean value of phase difference and the second mean value of phase difference are both mean value of phase difference, but they are not the same mean value of phase difference. The first image feature can be referred to as the second image feature, and similarly, the second image feature can be referred to as the first image feature. Both the first image feature and the second image feature are image features, but they are not the same image feature.

The embodiment of the present application provides an electronic device. For ease of description, only the parts related to the embodiments of the present application are shown. For specific technical details that are not disclosed, please refer to the method part of the embodiments of the present application. The electronic device can be any terminal device including mobile phone, tablet computer, PDA (Personal Digital Assistant), POS (Point of Sales), on-board computer, wearable device, etc. Take the electronic device as a mobile phone as an example . The above-mentioned electronic equipment includes an image processing circuit, which can be implemented by hardware and/or software components, and can include various processing units that define an ISP (Image Signal Processing, image signal processing) pipeline. Fig. 1 is a schematic diagram of an image processing circuit in an embodiment. As shown in FIG. 1, for ease of description, only various aspects of the image processing technology related to the embodiments of the present application are shown.

As shown in FIG. 1, the image processing circuit includes an ISP processor 140 and a control logic 150. The image data captured by the imaging device 110 is first processed by the ISP processor 140, and the ISP processor 140 analyzes the image data to capture image statistics that can be used to determine and/or one or more control parameters of the imaging device 110. The imaging device 110 may include a camera having one or more lenses 112 and an image sensor 114. The image sensor 114 may include a color filter array (such as a Bayer filter). The image sensor 114 may obtain the light intensity and wavelength information captured by each imaging pixel of the image sensor 114, and provide a set of raw materials that can be processed by the ISP processor 140. Image data. The attitude sensor 120 (such as a three-axis gyroscope, a Hall sensor, and an accelerometer) can provide the collected image processing parameters (such as anti-shake parameters) to the ISP processor 140 based on the interface type of the attitude sensor 120. The interface of the attitude sensor 120 may use an SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the foregoing interfaces.

In addition, the image sensor 114 may also send the original image data to the posture sensor 120. The sensor 120 can provide the original image data to the ISP processor 140 based on the interface type of the posture sensor 120, or the posture sensor 120 can store the original image data in the image memory 130. in.

The ISP processor 140 processes the original image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 140 may perform one or more image processing operations on the original image data, and collect statistical information about the image data. Among them, the image processing operations can be performed with the same or different bit depth accuracy.

The ISP processor 140 may also receive image data from the image memory 130. For example, the posture sensor 120 interface sends the original image data to the image memory 130, and the original image data in the image memory 130 is then provided to the ISP processor 140 for processing. The image memory 130 may be a part of a memory device, a storage device, or an independent dedicated memory in an electronic device, and may include DMA (Direct Memory Access) features.

When receiving raw image data from the image sensor 114 interface or from the posture sensor 120 interface or from the image memory 130, the ISP processor 140 may perform one or more image processing operations, such as temporal filtering. The processed image data can be sent to the image memory 130 for additional processing before being displayed. The ISP processor 140 receives the processed data from the image memory 130, and performs image data processing in the original domain and in the RGB and YCbCr color spaces on the processed data. The image data processed by the ISP processor 140 may be output to the display 160 for viewing by the user and/or further processed by a graphics engine or a GPU (Graphics Processing Unit, graphics processor). In addition, the output of the ISP processor 140 can also be sent to the image memory 130, and the display 160 can read image data from the image memory 130. In one embodiment, the image memory 130 may be configured to implement one or more frame buffers.

The statistical data determined by the ISP processor 140 may be sent to the control logic 150 unit. For example, the statistical data may include image sensor 114 statistical information such as the vibration frequency of the gyroscope, automatic exposure, automatic white balance, automatic focus, flicker detection, black level compensation, and lens 112 shadow correction. The control logic 150 may include a processor and/or a microcontroller that executes one or more routines (such as firmware). The one or more routines can determine the control parameters and ISP processing of the imaging device 110 based on the received statistical data. The control parameters of the device 140. For example, the control parameters of the imaging device 110 may include attitude sensor 120 control parameters (such as gain, integration time of exposure control, anti-shake parameters, etc.), camera flash control parameters, camera anti-shake displacement parameters, lens 112 control parameters (such as focus or Zoom focal length), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (for example, during RGB processing), and lens 112 shading correction parameters.

In one embodiment, the image sensor 114 in the imaging device (camera) may include a plurality of pixel point groups arranged in an array, wherein each pixel point group includes a plurality of pixel points arranged in an array, and each pixel point includes Multiple sub-pixels arranged in an array.

The first image is acquired through the lens 112 and the image sensor 114 in the imaging device (camera) 110, and the first image is sent to the ISP processor 140. After receiving the first image, the ISP processor 140 can perform subject detection on the first image to obtain the region of interest in the first image, or obtain the region selected by the user as the region of interest, or obtain it in other ways The area of interest is not limited to this.

The ISP processor 140 is configured to obtain a preview image, divide the preview image into at least two sub-areas, obtain a phase difference corresponding to each of the at least two sub-areas, and determine at least two target phase differences according to the phase difference corresponding to each sub-area , The at least two target phase differences include the target foreground phase difference and the target background phase difference, focus is performed according to each target phase difference, and an image corresponding to each target phase difference is obtained, and the image corresponding to each target phase difference is synthesized, Obtain a fully in-focus image. The ISP processor 140 may send relevant information of the target sub-region, such as location information, contour information, etc., to the control logic 150.

After receiving the relevant information of the target area, the control logic 150 controls the lens 112 in the imaging device (camera) to move, so as to focus on the position in the actual scene corresponding to the target area.

Fig. 2 is a flowchart of an image processing method in an embodiment. As shown in FIG. 2, an image processing method applied to an electronic device includes operation 202 to operation 212.

In operation 202, a preview image is obtained.

Among them, the number of cameras of the electronic device is not limited. For example, it may be one or two... and it is not limited to this. The form of the camera installed in the electronic device is not limited. For example, it can be a camera built into the electronic device, or it can be an external camera of the electronic device. It can be a front camera or a rear camera. The camera on the electronic device can be any type of camera. For example, the camera may be a color camera, a black-and-white camera, a depth camera, a telephoto camera, a wide-angle camera, etc., but is not limited thereto.

The preview image may be a visible light image. The preview image refers to the image presented on the screen of the electronic device when the camera is not shooting. The preview image can be the preview image of the current frame.

Specifically, the electronic device obtains a preview image through a camera and displays it on the display screen.

In operation 204, the preview image is divided into at least two sub-areas.

Among them, the sub-region refers to an image region in the preview image. The sub-region is a part of the image. That is, the sub-region includes a part of the pixels of the preview image. The size and shape of each sub-region obtained by dividing the preview image may be the same or different, and one of them may be the same and the other may be different. The specific division method is not limited.

Specifically, the electronic device divides the preview image into at least two sub-areas. The electronic device may divide the preview image into M×N sub-areas. Both N and M are positive integers, and the values of N and M may be the same or different. For example, the pixel of the preview image is 100×100, and it is divided into 4 sub-regions, then the pixels of each sub-region are 25×25.

Operation 206: Obtain a phase difference corresponding to each of the at least two sub-regions.

Among them, the phase difference refers to the difference in the position of the image formed by the imaging light entering the lens from different directions in the image sensor.

Specifically, the electronic device includes an image sensor, the image sensor may include a plurality of pixel point groups arranged in an array, each pixel point group includes M*N pixel points arranged in an array; each pixel point corresponds to a photosensitive unit, Among them, both M and N are natural numbers greater than or equal to 2. Then, the phase difference corresponding to each sub-region may include the first phase difference and the second phase difference. The first direction corresponding to the first phase difference and the second direction corresponding to the second phase difference form a preset angle. Wherein, the preset included angle may be any included angle other than 0 degrees, 180 degrees, and 360 degrees. That is, the phase difference corresponding to each sub-region can be two. The electronic device acquires the credibility of the first phase difference and the credibility of the second phase difference; determines the credibility of the credibility of the second phase difference and the credibility of the credibility of the second phase difference; The phase difference with higher reliability is used as the phase difference corresponding to the sub-region.

In one embodiment, in order to perform phase detection auto-focusing, usually some phase detection pixels can be arranged in pairs in the pixels included in the image sensor, which can also be called shielded pixels, where each phase detection pixel is paired One phase detection pixel point is occluded on the left side, and the other phase detection pixel point is occluded on the right side. In this way, the imaging beam directed at each phase detection pixel point pair can be separated into two parts, left and right, by comparing the left and right parts. The phase difference corresponding to each sub-window can be obtained from the image formed by the two imaging beams.

In operation 208, at least two target phase differences are determined from the phase differences corresponding to each sub-region, and the at least two target phase differences include the target foreground phase difference and the target background phase difference.

Among them, the foreground refers to the part with smaller depth in the image. The foreground contains the subject. The foreground is generally the object that the user wants to focus on. The at least two target phase differences include the foreground phase difference and the background phase difference, and may also include other phase differences. For example, the phase difference between the foreground and the background.

Specifically, the electronic device determines at least two target phase differences from the phase differences corresponding to each area, and the at least two target phase differences include at least the target foreground phase difference and the target background phase difference.

In operation 210, focusing is performed according to the phase difference of each target, and an image corresponding to the phase difference of each target is obtained.

Among them, focusing refers to the process of changing the distance and distance of the object through the focusing mechanism of the electronic device to make the image of the object clear. Focus can refer to auto focus. Auto focus may refer to Phase Detection Auto Focus (PDAF) and other auto focus methods combined with phase focus. Phase focusing is to obtain the phase difference through the sensor, calculate the defocus value according to the phase difference, and control the lens to move the corresponding distance according to the defocus value to achieve focus. Phase focus can be combined with other focus methods, such as continuous auto focus, laser focus, etc.

Specifically, when receiving a photographing instruction, the electronic device performs focusing according to each target phase difference of the at least two target phase differences, and obtains an image corresponding to each target phase difference. The electronic device can calculate the defocus value corresponding to each target phase difference according to each target phase difference of at least two target phase differences, and control the lens to move the corresponding distance according to each defocus value to obtain the corresponding distance of each target phase difference. image. The defocus value refers to the distance between the current position of the image sensor and the position where the image sensor should be in the in-focus state. Each phase difference has a corresponding defocus value. The defocus value corresponding to each phase difference can be the same or different. The relationship between the phase difference and the defocus value can be obtained by pre-calibration. For example, the relationship between the phase difference and the defocus value may be a linear relationship or a nonlinear relationship.

For example, the at least two target phase differences include target phase difference A, target phase difference B, and target phase difference C. Among them, the target phase difference A is the target foreground phase difference, the target phase difference C is the target background phase difference, and the target phase difference B is the phase difference between the target phase difference A and the target phase difference C. The defocus value A is calculated according to the target phase difference A, and the lens is moved by the corresponding distance according to the defocus value A to obtain the image A corresponding to the target phase difference A. The defocus value B is calculated according to the target phase difference B, and the lens is controlled to move the corresponding distance according to the defocus value B to obtain the image B corresponding to the target phase difference B. The defocus value C is calculated according to the target phase difference C, and the lens is moved by the corresponding distance according to the defocus value C to obtain the image C corresponding to the target phase difference C. Then the electronic device gets image A, image B, and image C. The processing order of the target phase difference is not limited.

In operation 212, the image corresponding to each target phase difference is synthesized to obtain a fully in-focus image.

Among them, the fully in-focus image refers to an image in which there is no out-of-focus area theoretically. Image stitching refers to combining several images, which can be obtained by focusing at different positions, or images corresponding to different phase differences, to form a seamless panoramic image or high-resolution image.

Specifically, the electronic device may stitch and synthesize the clear parts of the image corresponding to each target phase difference to obtain a fully in-focus image. Alternatively, the electronic device may use the Laplacian pyramid method to perform the process according to the image corresponding to the phase difference of each target to obtain a fully in-focus image. Alternatively, the electronic device inputs the image corresponding to the phase difference of each target into the convolutional neural network model for synthesis to obtain a fully in-focus image and the like is not limited to this.

In the image processing method in this embodiment, a preview image is obtained, the preview image is divided into at least two sub-regions, the phase difference corresponding to each sub-region in the at least two sub-regions is obtained, and at least two are determined from the phase difference corresponding to each sub-region. Target phase difference, at least two target phase differences include the target foreground phase difference and the target background phase difference. Focusing is performed according to the phase difference of each target to obtain an image corresponding to the phase difference of each target, and at least two targets at different focal points can be acquired. One is the background in-focus image, and the other is the foreground in-focus image. The images corresponding to the phase difference of each target are synthesized to obtain a fully in-focus image, which can obtain an image with less out-of-focus area and improve the image The clarity.

In an embodiment, FIG. 3 is a schematic diagram of the principle of phase focusing in an embodiment. M1 is the position of the image sensor when the electronic device is in the in-focus state. The in-focus state refers to the state of successful focusing. Please refer to Figure 3. When the image sensor is in the M1 position, the object W is reflected toward the lens Lens The imaging light g in different directions of the image converges on the image sensor, that is, the imaging light g in different directions reflected by the object W toward the lens Lens is imaged at the same position on the image sensor. At this time, the image of the image sensor is clear .

M2 and M3 are the possible positions of the image sensor when the electronic device is not in focus. As shown in Figure 3, when the image sensor is at the M2 position or the M3 position, the object W is reflected toward the lens Lens in different directions. The imaging light g will be imaged at different positions. Please refer to Figure 3, when the image sensor is at the M2 position, the imaging light g reflected by the object W in different directions to the lens Lens is imaged at the position A and the position B respectively. When the image sensor is at the M3 position, the object W is reflected toward The imaging rays g in different directions of the lens Lens are respectively imaged at the position C and the position D. At this time, the image of the image sensor is not clear.

In PDAF technology, the difference in position of the image formed by the imaging light entering the lens from different directions in the image sensor can be obtained. For example, as shown in Figure 3, the difference between position A and position B can be obtained, or, Obtain the difference between position C and position D; after obtaining the difference in position of the image formed by the imaging light entering the lens from different directions in the image sensor, the difference and the difference between the lens and the image sensor in the camera The geometric relationship is used to obtain the defocus value. The so-called defocus value refers to the distance between the current position of the image sensor and the position where the image sensor should be in the in-focus state; the electronic device can focus according to the obtained defocus value.

Among them, the "difference in the position of the image formed by the imaging light entering the lens from different directions on the image sensor" can generally be referred to as a phase difference. According to the above description, in PDAF technology, obtaining the phase difference is a very critical technical link.

It should be pointed out that in practical applications, the phase difference can be applied to a variety of different scenes, and the focus scene is only a relatively possible scene. For example, the phase difference can be applied to the acquisition scene of the depth map, that is, the phase difference can be used to acquire the depth map; for another example, the phase difference can be used in the reconstruction scene of the three-dimensional image, that is, you can use The phase difference realizes the reconstruction of the three-dimensional image. The embodiment of the present application aims to provide a method for obtaining the phase difference. As for the scene to which the phase difference is applied after the phase difference is obtained, the embodiment of the present application does not specifically limit it.

In the related art, some phase detection pixels may be provided in pairs among the pixels included in the image sensor, please refer to FIG. 4. FIG. 4 is a schematic diagram of phase detection pixel points arranged in pairs in the pixel points included in the image sensor in an embodiment. As shown in FIG. 4, a phase detection pixel point pair (hereinafter referred to as a pixel point pair) A, a pixel point pair B, and a pixel point pair C may be provided in the image sensor. Among them, in each pixel point pair, one phase detection pixel is subjected to left shielding (English: Left Shield), and the other phase detection pixel is subjected to right shielding (English: Right Shield).

For the phase detection pixel point that has been blocked on the left, only the right beam of the imaging beam directed to the phase detection pixel point can be in the photosensitive part of the phase detection pixel point (that is, the part that is not blocked). ). For the phase detection pixel that has been occluded on the right, only the left beam of the imaging beam directed at the phase detection pixel can be in the photosensitive part of the phase detection pixel (that is, not The occluded part) is imaged. In this way, the imaging beam can be divided into left and right parts, and the phase difference can be obtained by comparing the images formed by the left and right parts of the imaging beam. The focusing method using the Figure 4 method is to obtain the phase difference through the sensor, calculate the defocus value according to the phase difference, control the lens movement according to the defocus value, and then find the focus value (FV) peak value.

In one embodiment, determining at least two target phase differences from the phase differences corresponding to each sub-region, and the at least two target phase differences include the target foreground phase difference and the target background phase difference, including:

In operation (a1), the phase difference corresponding to each of the at least two sub-regions is divided into a foreground phase difference set and a background phase difference set.

Wherein, the foreground phase difference set includes at least one foreground phase difference. The background phase difference set includes at least one background phase difference.

Specifically, the phase difference threshold may be stored in the electronic device. The phase difference greater than the phase difference threshold is divided into a background phase difference set, and the phase difference less than or equal to the phase difference threshold is divided into a foreground phase difference set.

In this embodiment, the electronic device calculates the median phase difference according to the phase difference corresponding to each sub-region. The phase difference greater than the median of the phase difference is divided into the background phase difference set, and the phase difference less than or equal to the median of the phase difference is divided into the foreground phase difference set.

Operation (a2) is to obtain the first mean value of the phase difference corresponding to the foreground phase difference set.

Specifically, the electronic device obtains an average value according to the phase difference in the foreground phase difference set to obtain the first average value of the phase difference.

Operation (a3) is to obtain the second mean value of phase difference corresponding to the background phase difference set.

Specifically, the electronic device obtains the average value according to the phase difference in the background phase difference set to obtain the second average value of the phase difference.

In operation (a4), the first average value of the phase difference is used as the target foreground phase difference.

Specifically, regardless of whether the first average value of phase difference is the same as any one of the at least two sub-regions, the electronic device uses the first average value of phase difference as the target foreground phase difference.

When the first average value of phase difference is different from any phase difference in the at least two sub-regions, the corresponding first defocus value is calculated according to the first average value of phase difference, and the lens is controlled to move the corresponding distance according to the first defocus value to obtain The image corresponding to the first mean value of the phase difference.

When the first average value of phase difference is the same as any one of the at least two sub-areas, the area corresponding to the first average value of phase difference is used as the focus area for focusing, and an image corresponding to the first average value of phase difference is obtained.

In operation (a5), the second average phase difference is used as the target background phase difference.

Specifically, regardless of whether the second average value of the phase difference is the same as any one of the at least two sub-regions, the electronic device uses the second average value of the phase difference as the target foreground phase difference.

When the second average value of phase difference is different from any phase difference in the at least two sub-regions, the corresponding second defocus value is calculated according to the second average value of phase difference, and the lens is controlled to move the corresponding distance according to the second defocus value to obtain The image corresponding to the second mean value of the phase difference.

When the second average value of phase difference is the same as any one of the at least two sub-areas, the area corresponding to the second average value of phase difference is used as the focus area for focusing to obtain an image corresponding to the second average value of phase difference.

In the image processing method in this embodiment, the phase difference corresponding to each of the at least two sub-regions is divided into a foreground phase difference set and a background background phase difference set, and the first phase difference average value corresponding to the foreground phase difference set is obtained, and The second average value of phase difference corresponding to the background phase difference set of the background, the first average value of phase difference is taken as the target foreground phase difference, and the second average value of phase difference is taken as the target background phase difference, the foreground in-focus image and background in-focus image can be obtained according to the average value Image, improve the clarity of the image.

In an embodiment, the image processing method further includes: excluding the maximum phase difference among the phase differences corresponding to the sub-regions to obtain a residual phase difference set; and dividing the phase difference corresponding to each sub-region in the at least two sub-regions into the foreground phase difference The set and the background phase difference set include: dividing the remaining phase difference set into a foreground phase difference set and a background background phase difference set.

Specifically, the area corresponding to the largest phase difference among the phase differences corresponding to the sub-regions is the area corresponding to the farthest scene in the preview image. Excluding the maximum phase difference among the phase differences corresponding to the sub-regions is to exclude the phase difference corresponding to the farthest scene in the preview image.

The phase difference threshold can be stored in the electronic device. The phase difference that is greater than the phase difference threshold in the remaining phase difference set is divided into a background phase difference set, and the phase difference that is less than or equal to the phase difference threshold is divided into a foreground phase difference set.

In this embodiment, the electronic device calculates the median phase difference according to the phase difference in the remaining phase difference set. The phase difference greater than the median of the phase difference is divided into the background phase difference set, and the phase difference less than or equal to the median of the phase difference is divided into the foreground phase difference set.

In this embodiment, the electronic device calculates the average value of the phase difference according to the phase difference in the remaining phase difference set. The phase difference greater than the average phase difference is divided into a background phase difference set, and the phase difference less than or equal to the average phase difference is divided into a foreground phase difference set.

In the image processing method in this embodiment, since the furthest background details are often not important, the largest phase difference in the phase difference corresponding to the sub-region is excluded to obtain the remaining phase difference set, which can eliminate the farthest background and reduce the remaining phase difference. The set is divided into a foreground phase difference set and a background background phase difference set. Focusing based on the average value can improve the clarity of the image.

In one embodiment, determining at least two target phase differences from the phase differences corresponding to each sub-region, and the at least two target phase differences include the foreground phase difference and the background phase difference, including: obtaining the phase difference of the at least two sub-regions The maximum phase difference and the minimum phase difference; the minimum phase difference is regarded as the foreground phase difference; the maximum phase difference is regarded as the background phase difference.

Among them, the area corresponding to the largest phase difference is the area corresponding to the most distant object. The area corresponding to the smallest phase difference is the area corresponding to the nearest scene. There is usually a target subject in the area corresponding to the smallest phase difference.

The image processing method in this embodiment obtains the maximum phase difference and the minimum phase difference among the phase differences of at least two sub-regions, and uses the minimum phase difference as the foreground phase difference and the maximum phase difference as the background phase difference, so that only two The image is synthesized to improve image processing efficiency while improving image clarity.

In one embodiment, the electronic device includes an image sensor, the image sensor includes a plurality of pixel point groups arranged in an array, and each pixel point group includes M*N pixel points arranged in an array; each pixel point corresponds to a photosensitive unit , Where both M and N are natural numbers greater than or equal to 2.

In this embodiment, FIG. 5 is a schematic diagram of a part of the structure of an electronic device in an embodiment. As shown in FIG. 5, the electronic device may include a lens 502 and an image sensor 504, where the lens 502 may be composed of a series of lenses, and the image sensor 504 may be a metal oxide semiconductor device (English: Complementary Metal Oxide Semiconductor; abbreviation: CMOS) ) Image sensor, charge-coupled device (English: Charge-coupled Device; abbreviation: CCD), quantum thin film sensor or organic sensor, etc.

Please refer to FIG. 6, which shows a schematic structural diagram of a part of the image sensor 504. As shown in FIG. 6, the image sensor 504 may include a plurality of pixel point groups Z arranged in an array, wherein each pixel point group Z includes There are a plurality of pixel points D arranged in an array, and each pixel point includes a plurality of sub-pixel points d arranged in an array. Please refer to FIG. 6, optionally, each pixel point group Z may include 4 pixels D arranged in an array arrangement of two rows and two columns, and each pixel point may include an array of two rows and two columns. 4 sub-pixel points d arranged in an arrangement manner.

It should be pointed out that the pixel points included in the image sensor 504 refer to a photosensitive unit, which may be composed of a plurality of photosensitive elements (that is, sub-pixels) arranged in an array, wherein the photosensitive element is a kind of An element that converts optical signals into electrical signals. Optionally, the photosensitive unit may further include a microlens and a filter, etc., wherein the microlens is disposed on the filter, the filter is disposed on each photosensitive element included in the photosensitive unit, and the filter may include red , Green, and blue, which can only transmit light of the corresponding wavelengths of red, green, and blue respectively.

FIG. 7 is a schematic diagram of the structure of pixels in an embodiment. As shown in Figure 7, taking each pixel point including sub-pixel point 1, sub-pixel point 2, sub-pixel point 3, and sub-pixel point 4 as an example, sub-pixel point 1 and sub-pixel point 2 can be combined, and sub-pixel point 3 and sub-pixel point Pixel 4 is synthesized to form a pair of PD pixels in the vertical direction to obtain the phase difference in the vertical direction, which can detect horizontal edges; sub-pixel point 1 and sub-pixel point 3 are synthesized, and sub-pixel point 2 and sub-pixel point 4 are synthesized to form a left-right direction. The PD pixel pair obtains the phase difference in the horizontal direction and can detect the vertical edge.

FIG. 8 is a schematic diagram of the internal structure of an image sensor in an embodiment. The imaging device includes a lens and an image sensor. As shown in FIG. 8, the image sensor includes a micro lens 80, a filter 82 and a photosensitive unit 84. The micro lens 80, the filter 82 and the photosensitive unit 84 are sequentially located on the incident light path, that is, the micro lens 80 is disposed on the filter 82, and the filter 82 is disposed on the photosensitive unit 84.

The filter 82 may include three types of red, green, and blue, and can only transmit light of corresponding wavelengths of red, green, and blue, respectively. One filter 82 is arranged on one pixel point.

The micro lens 80 is used to receive incident light and transmit the incident light to the filter 82. After the filter 82 smoothes the incident light, the smoothed light is incident on the photosensitive unit 84 on a pixel basis.

The photosensitive unit 84 in the image sensor converts the light incident from the filter 82 into a charge signal through the photoelectric effect, and generates a pixel signal consistent with the charge signal. The charge signal is consistent with the received light intensity.

It can be seen from the above description that the pixels included in the image sensor and the pixels included in the image are two different concepts. The pixels included in the image refer to the smallest component unit of the image, which is generally represented by a sequence of numbers. In the following, the sequence of numbers can be referred to as the pixel value of a pixel. The embodiments of the present application involve both concepts of "pixels included in an image sensor" and "pixels included in an image". To facilitate readers' understanding, a brief explanation is provided here.

Please refer to FIG. 9, which shows a schematic diagram of an exemplary pixel point group Z. As shown in FIG. 9, the pixel point group Z includes 4 pixels arranged in an array arrangement of two rows and two columns. D, where the color channel of the pixels in the first row and the first column is green, that is, the filters included in the pixels in the first row and the first column are green filters, and the pixels in the first row and the second column are green filters. The color channel of the pixel is red, that is, the filter included in the pixel in the first row and second column is a red filter, and the color channel of the pixel in the second row and first column is blue, that is, Yes, the pixel in the second row and first column includes a blue filter, and the color channel of the pixel in the second row and second column is green, that is, the pixel in the second row and second column The filter included in the dot is a green filter.

In one embodiment, the electronic device includes an image sensor, and the image sensor includes a plurality of pixel point groups arranged in an array, and each pixel point group includes a plurality of pixel points arranged in an array. As shown in FIG. 10, it is a schematic diagram of the process of obtaining the phase difference corresponding to each sub-region in an embodiment, and obtaining the phase difference corresponding to each sub-region in at least two sub-regions includes:

In operation 1002, a target brightness map is obtained according to the brightness values of the pixel points included in each pixel point group.

Wherein, the brightness value of the pixel of the image sensor can be characterized by the brightness value of the sub-pixel included in the pixel. In other words, the electronic device can obtain the target brightness map according to the brightness values of the sub-pixels in the pixel points included in each pixel point group. The "brightness value of a sub-pixel" refers to the brightness value of the light signal received by the sub-pixel.

The sub-pixel included in the image sensor is a photosensitive element that can convert light signals into electrical signals. Therefore, the electronic device can obtain the intensity of the light signal received by the sub-pixel according to the electrical signal output by the sub-pixel, and obtain the brightness value of the sub-pixel according to the intensity of the light signal received by the sub-pixel.

The target brightness map in the embodiment of the present application is used to reflect the brightness value of the sub-pixels in the image sensor. The target brightness map may include multiple pixels, wherein the pixel value of each pixel in the target brightness map is based on the image sensor Obtained from the brightness value of the neutron pixel.

In operation 1004, segmentation processing is performed on the target brightness map, and the first segmented brightness map and the second segmented brightness map are obtained according to the results of the segmentation processing.

Specifically, the electronic device may perform segmentation processing on the target brightness map along the direction of the column (the y-axis direction in the image coordinate system). The first segmented brightness map and the second segmented brightness map obtained after the target brightness map is segmented along the column direction can be called the left image and the right image, respectively.

In this embodiment, the electronic device may perform segmentation processing on the target brightness map along the row direction (the x-axis direction in the image coordinate system). The first segmented brightness map and the second segmented brightness map obtained after the target brightness map is segmented in the direction of the row can be referred to as the upper image and the lower image, respectively.

In operation 1006, the phase difference of the pixels that match each other is determined according to the position difference of the pixels that match each other in the first split brightness map and the second split brightness map.

Specifically, taking the segmentation processing of the target brightness map along the direction of the row (the x-axis direction in the image coordinate system) as an example, the obtained first brightness segmentation map and the second brightness segmentation map are the upper and lower images. Figure. Then, the electronic device obtains the vertical phase difference according to the position difference of the matching pixels in the first brightness segmentation map and the second brightness segmentation map.

Taking the segmentation processing of the target brightness map along the column direction (the y-axis direction in the image coordinate system) as an example, the obtained first brightness segmentation map and the second brightness segmentation map are the left and right images. Then, the electronic device obtains the horizontal phase difference according to the position difference of the matching pixels in the first brightness segmentation map and the second brightness segmentation map.

Among them, "mutually matched pixels" means that the pixel matrix composed of the pixel itself and the surrounding pixels are similar to each other. For example, the pixel a and its surrounding pixels in the first segmented brightness map form a pixel matrix with 3 rows and 3 columns, and the pixel value of the pixel matrix is:

2 10 90

1 20 80

0 100 1

The pixel b and the surrounding pixels in the second segmented brightness map also form a pixel matrix with 3 rows and 3 columns, and the pixel value of the pixel matrix is:

1 10 90

1 21 80

0 100 2

It can be seen from the above that the two matrices are similar, and it can be considered that the pixel a and the pixel b match each other. As for how to judge whether the pixel matrix is similar, there are many different methods in practical applications. A common method is to calculate the difference of the pixel value of each corresponding pixel in the two pixel matrices, and then calculate the difference. The absolute value of is added, and the result of the addition is used to determine whether the pixel matrix is similar, that is, if the result of the addition is less than a preset threshold, the pixel matrix is considered similar, otherwise, the pixel matrix is considered not similar.

For example, for the above two pixel matrices with 3 rows and 3 columns, the difference of 1 and 2, the difference of 10 and 10, the difference of 90 and 90, ..., and then the absolute difference Values are added, and the result of the addition is 3. If the result of the addition of 3 is less than the preset threshold, it is considered that the two pixel matrices with 3 rows and 3 columns are similar.

Another common method for judging whether the pixel matrix is similar is to use the Sobel convolution kernel calculation method or the high Laplacian calculation method to extract the edge characteristics, and judge whether the pixel matrix is similar by the edge characteristics.

In this embodiment, "the positional difference of pixels that match each other" refers to the difference between the positions of the pixels in the first split brightness map and the positions of the pixels in the second split brightness map among the matched pixels. . As in the above example, the position difference between the pixel a and the pixel b that are matched with each other refers to the difference between the position of the pixel a in the first split brightness map and the position of the pixel b in the second split brightness map.

The pixels that match each other correspond to different images in the image sensor formed by the imaging light entering the lens from different directions. For example, the pixel a in the first split brightness map and the pixel b in the second split brightness map match each other, where the pixel a may correspond to the image formed at position A in FIG. 1, and the pixel b may correspond to The image formed at position B in Figure 1.

Since the matched pixels correspond to the different images in the image sensor formed by the imaging light entering the lens from different directions, the phase difference of the matched pixels can be determined according to the position difference of the matched pixels. .

In operation 1008, the phase difference corresponding to each of the at least two sub-regions is determined according to the phase difference of the pixels that are matched with each other.

Specifically, the electronic device determines a phase difference corresponding to each of the at least two sub-regions according to the mutually matched phase differences.

In this embodiment, the electronic device can obtain two phase differences corresponding to each sub-region according to the phase differences of the pixels that are matched with each other, which are the vertical phase difference and the horizontal phase difference, respectively. The electronic device can obtain the vertical phase difference confidence level and the horizontal phase difference confidence level corresponding to each sub-region, determine the phase difference with the highest confidence level, and use the phase difference as a phase difference corresponding to each sub-region.

The image processing method in this embodiment obtains a target brightness map according to the brightness values of the pixel points included in each pixel point group in the image sensor. After the target brightness map is obtained, the target brightness map is segmented and processed according to As a result of the segmentation processing, the first segmented brightness map and the second segmented brightness map are obtained. Then, based on the position difference of the matching pixels in the first segmented brightness map and the second segmented brightness map, the pixels that match each other are determined Then, according to the phase difference of the matched pixels, the phase difference corresponding to each of the at least two sub-regions is determined. In this way, the brightness value of the pixel points included in each pixel point group in the image sensor can be used to determine the phase difference. Obtain the target phase difference map. Therefore, compared with the method of obtaining phase difference by using sparsely arranged phase detection pixels, the phase difference of the matched pixels in the embodiment of the present application contains relatively rich phase difference information. The acquired phase difference accuracy can be improved, so that when focusing, a high-precision phase difference corresponding to the focus area can be obtained, the focus peak can not be found, the focus efficiency is improved, and the overall in-focus image synthesis efficiency is improved.

In one embodiment, segmentation processing is performed on the target brightness map, and the first segmented brightness map and the second segmented brightness map are obtained according to the results of the segmentation processing, including:

Operation (b1), segmenting the target brightness map to obtain multiple brightness map regions, each brightness map region includes a row of pixels in the target brightness map, or each brightness map region includes a column of pixels in the target brightness map .

Wherein, each luminance map area includes a row of pixels in the target luminance map. Alternatively, each luminance map area includes a column of pixels in the target luminance map.

In this embodiment, the electronic device may segment the target brightness map column by column along the row direction to obtain multiple pixel columns of the target brightness map.

In this embodiment, the electronic device may segment the target brightness map row by row along the column direction to obtain multiple pixel rows of the target brightness map.

Operation (b2), obtain a plurality of first brightness map regions and a plurality of second brightness map regions from a plurality of brightness map regions, where the first brightness map region includes pixels in even rows of the target brightness map, or the first brightness map The area includes pixels in even-numbered columns in the target luminance map, and the second luminance map area includes pixels in odd-numbered rows in the target luminance map, or the second luminance map area includes pixels in odd-numbered columns in the target luminance map.

Wherein, the first luminance map area includes pixels in even rows of the target luminance map. Alternatively, the first luminance map area includes pixels in even-numbered columns in the target luminance map.

The second luminance map area includes pixels in odd rows in the target luminance map. Or, the second luminance map area includes pixels in odd-numbered columns in the target luminance map.

In this embodiment, when the target brightness map is segmented column by column, the electronic device may determine the even-numbered columns as the first brightness map area, and the odd-numbered columns as the second brightness map area.

In this embodiment, when the target brightness map is segmented row by row, the electronic device may determine even-numbered rows as the first brightness map area, and odd-numbered rows as the second brightness map area.

Operation (b3) is to use a plurality of first brightness map regions to form a first segmented brightness map, and use a plurality of second brightness map regions to form a second segmented brightness map.

In this embodiment, FIG. 11 is a schematic diagram of performing segmentation processing on the target brightness map in the first direction in an embodiment. FIG. 12 is a schematic diagram of performing segmentation processing on the target brightness map in the second direction in an embodiment. As shown in FIG. 11, assuming that the target brightness map includes 6 rows and 6 columns of pixels, when the target brightness map is segmented column by column, that is, the target brightness map is segmented in the first direction. The electronic device can determine the pixels in the first column, the pixels in the third column, and the pixels in the fifth column of the target brightness map as the first brightness map area, and can set the pixels in the second column, the fourth column and the sixth column of the target brightness map. Determined as the second brightness map area. Then, the electronic device may splice the first brightness map area to obtain a first split brightness map T1, which includes the first column, the third column, and the fifth column of the target brightness map. . The electronic device may splice the second brightness map regions to obtain a second segmented brightness map T2, which includes the second column of pixels, the fourth column of pixels, and the sixth column of pixels of the target brightness map.

As shown in FIG. 12, assuming that the target brightness map includes 6 rows and 6 columns of pixels, when the target brightness map is segmented row by row, that is, the target brightness map is segmented in the second direction. The electronic device can determine the pixels in the first row, the pixels in the third row, and the pixels in the fifth row of the target brightness map as the first brightness map area, and can set the pixels in the second row, the fourth row and the sixth row of the target brightness map. Determined as the second brightness map area, and then, the electronic device can splice the first brightness map area to obtain the first sub-brightness map T3. The first sub-brightness map T3 includes the pixels in the first row and the first row of the target brightness map. 3 rows of pixels and 5th row of pixels. The electronic device may splice the second brightness map regions to obtain a second segmented brightness map T4, which includes the second row of pixels, the fourth row of pixels, and the sixth row of pixels of the target brightness map.

The image processing method in the embodiment of the present application does not need to block pixels to obtain the phase difference, and obtains relatively rich phase difference information by means of brightness segmentation, which improves the accuracy of the obtained phase difference.

In an embodiment, the electronic device includes an image sensor, the image sensor includes a plurality of pixel point groups arranged in an array, each of the pixel point groups includes M*N pixel points arranged in an array; each pixel point corresponds to one In the photosensitive unit, both M and N are natural numbers greater than or equal to 2. The phase difference corresponding to each sub-region includes a horizontal phase difference and a vertical phase difference. Obtain the phase difference corresponding to each sub-region in at least two sub-regions, including: when the sub-region is detected to contain horizontal lines, the vertical phase difference is regarded as the phase difference corresponding to the sub-region; when the sub-region does not contain horizontal lines , Regard the horizontal phase difference as the phase difference corresponding to the sub-region.

Specifically, due to some characteristics of the camera, there will be a difference in imaging. That is, the lines may have problems such as smearing. Then when it is detected that the sub-region contains horizontal lines, the vertical phase difference is used as the phase difference corresponding to the sub-region; when it is detected that the sub-region contains vertical lines, the horizontal phase difference is used as the phase difference corresponding to the sub-region, which can improve the phase Poor acquisition accuracy, thereby improving image clarity.

In one embodiment, focusing according to the phase difference of each target to obtain an image corresponding to the phase difference of each target includes: taking the sub-area corresponding to the phase difference of each target as the focus area to obtain the image corresponding to the phase difference of each target .

Specifically, the electronic device uses a sub-area corresponding to each target phase difference of the at least two target phase differences as a focus area, and obtains an image corresponding to each target phase difference. For example, at least two target phase differences include target phase difference A, target phase difference B, and target phase difference C. Then, the sub-area corresponding to the target phase difference A is used as the focus area, and focus is performed to obtain an image corresponding to the target phase difference A. The sub-area corresponding to the target phase difference B is used as the focus area, and focus is performed to obtain an image corresponding to the target phase difference B. The sub-region corresponding to the target phase difference C is used as the focus area, and focus is performed to obtain an image corresponding to the target phase difference C. That is, a total of three images are obtained.

The image processing method in this embodiment uses the sub-region corresponding to each target phase difference as a focus area to obtain an image corresponding to each target phase difference, and can obtain images with different focal points for synthesis, thereby improving image clarity.

In one embodiment, synthesizing the images corresponding to each target phase difference to obtain a fully in-focus image includes: dividing the image corresponding to each target phase difference into the same number of sub-image areas; obtaining the corresponding sub-image area According to the corresponding definition of each sub-image area, determine the sub-image area with the highest definition among the matching sub-image areas; stitch and combine the sub-image areas with the highest definition to obtain a fully in-focus image.

Among them, the sub-image areas that match each other refer to sub-image areas located at the same position in different images.

Specifically, the electronic device divides the image corresponding to each target phase difference into the same number of sub-image areas. The electronic device obtains the definition corresponding to each sub-image area in the image corresponding to each target phase difference. The electronic device determines the sub-image area with the highest definition among the matched sub-image areas according to the corresponding definition of each sub-image area of each sub-image. The electronic device synthesizes all the sub-image areas with the highest definition to obtain a fully in-focus image. For example, the target phase difference A corresponds to image A, and image A is divided into sub-image area 1, sub-image area 2, sub-image area 3, and sub-image area 4. The target phase difference B corresponds to image B, which is divided into sub-image area α, sub-image area β, sub-image area γ, and sub-image area δ. Among them, the sub-image area 1 is located at the upper left corner of the image A, and the sub-image area α is located at the upper left corner of the image B, then the sub-image area 1 matches the sub-image area α... and so on. If the sub-image area 1, sub-image area β, sub-image area γ, and sub-image area 4 with the highest definition, the electronic device will splice and synthesize sub-image area 1, sub-image area β, sub-image area γ, and sub-image area 4. Obtain a fully in-focus image.

In the image processing method in the embodiment of the present application, the image corresponding to each target phase difference is divided into the same number of sub-image areas; the definition corresponding to each sub-image area is obtained; and the mutual definition is determined according to the corresponding definition of each sub-image area. The sub-image area with the highest definition among the matched sub-image areas; the sub-image area with the highest definition is synthesized to obtain a full in-focus image, which can quickly obtain a full-in-focus image and improve image processing efficiency.

In an embodiment, as shown in FIG. 13, it is a schematic flow chart of synthesizing to obtain a full in-focus image in an embodiment. Synthesize the images corresponding to the phase difference of each target to obtain a fully in-focus image, including:

Operation 1302, convolution and sampling processing of the image corresponding to each target phase difference, and when the preset iterative condition is met, a Gaussian pyramid of the image corresponding to each target phase difference is obtained.

Among them, the Gaussian pyramid is a kind of image pyramid. Except for the bottom layer image, the other layer images in the pyramid are all obtained by convolving and sampling the previous layer image. Gaussian pyramids can be used to obtain low-frequency images. The low-frequency image can refer to the contour image in the image. The iteration condition may mean reaching a preset number of times or reaching a preset time, etc., and is not limited thereto. The image corresponding to each target phase difference has a corresponding Gaussian pyramid. For example, the image A corresponding to the target phase difference corresponds to the Gaussian pyramid A, and the image B corresponding to the target phase difference corresponds to the Gaussian pyramid B.

Specifically, the electronic device uses a Gaussian kernel to convolve the image corresponding to each target phase difference, and sample the convolved image to obtain each layer of image. That is, the image (set as G0) corresponding to each target phase difference is convolved and sampled to obtain the upper low-frequency image (G1); then image G1 is convolved and sampled to obtain image G2...until the preset is satisfied The iterative condition, for example, iterates 5 times to obtain image G5, and obtains a Gaussian pyramid containing multiple low-frequency images corresponding to each target phase difference.

In operation 1304, processing is performed according to each layer of the image in the Gaussian pyramid of the image corresponding to each target phase difference to obtain the Laplacian pyramid of the image corresponding to each target phase difference.

Among them, in the calculation process of the Gaussian pyramid, some high-frequency detail information will be lost after the image undergoes convolution and down-sampling operations. To describe these high-frequency information, the Laplacian Pyramid (LP) is defined. The image corresponding to each target phase difference has a corresponding Laplacian pyramid. For example, the image A corresponding to the target phase difference corresponds to the Laplacian Pyramid A, and the image B corresponding to the target phase difference corresponds to the Laplacian Pyramid B. Each layer of the Laplace Pyramid represents different scales and details. Among them, the details can be regarded as frequency.

Specifically, the electronic device obtains the high-frequency image by subtracting the up-sampled low-frequency image from the original image. The formula can be L0=I0-G1, where L0 is the bottom layer of the Laplace pyramid. Thus, L1, L2, L3, L4... and then the Laplacian pyramid of the image corresponding to each target phase difference can be obtained.

In operation 1306, the Laplacian pyramid of the image corresponding to the phase difference of each target is fused to obtain a fused Laplacian pyramid.

Among them, there is only one Laplace Pyramid after fusion.

Specifically, the electronic device obtains the weight of the image corresponding to each target phase difference, and performs fusion according to the weight of the image corresponding to each target phase difference and the Laplacian pyramid of the image corresponding to each target phase difference, to obtain the fused image Pyramid of Laplace.

For example, the fusion formula is as follows:

L5(fusion)=Weight(picture 1)*L5(picture 1)+Weight(picture 2)*L5(picture 2)

Among them, L5 (fusion) refers to the sixth layer of the fused Laplacian pyramid from the bottom to the top. Weight (Figure 1) refers to the weight of Figure 1. Weight (Figure 2) refers to the weight of Figure 2. L5 (Figure 1) refers to the sixth layer of the Laplacian Pyramid in Figure 1 from the bottom up. L5 (Figure 2) refers to the sixth layer of the Laplacian Pyramid in Figure 2 counted from the bottom up.

Among them, the weight of each image can be adjusted according to parameters such as depth of field and degree of blur. For example, regions with a low degree of ambiguity are highly weighted. Areas with a high degree of blur have a small weight. Areas with a small depth of field are more powerful. The area with a large depth of field has a small weight.

In operation 1308, reconstruction processing is performed according to the fused Laplacian pyramid to obtain a fully in-focus image.

Specifically, electronic devices merge from the top to the bottom. The electronic device can process the top-level image in the fused Laplacian pyramid and Gaussian pyramid, that is, reconstruct it to obtain a fully in-focus image. For example, R5(fusion)=L5(fusion)+G5.

Among them, G5 can be obtained by fuse of the G5 layer image of the Gaussian pyramid corresponding to each target phase difference. L5 (fusion) is the L5 layer of the Laplace pyramid after fusion. R5 (fusion) is the G5 layer of the Laplace Pyramid after reconstruction (Reconstruction).

Then R5 (fusion) is up-sampled, then R4 = R5 (up-sampling) + L4 (fusion).

Among them, L4 (fusion) is the L4 layer of the Laplace pyramid after fusion. By analogy, the final synthesis result R0 can be obtained, which is the full in-focus image.

In the image processing method in this embodiment, the image convolution and sampling processing for each target phase difference is processed, and when the preset iterative conditions are met, the Gaussian pyramid of the image corresponding to each target phase difference is obtained, and the Gaussian pyramid is obtained according to each target phase difference. Each layer of the image in the Gaussian pyramid of the image corresponding to the difference is processed to obtain the Laplacian pyramid of the image corresponding to each target phase difference, and the Laplacian pyramid of the image corresponding to each target phase difference is merged to obtain The fused Laplacian Pyramid is reconstructed according to the fused Laplacian Pyramid to obtain a fully in-focus image. The image can be synthesized according to the low-frequency contour and high-frequency details to make the boundary between each area more Naturally, improve the authenticity and clarity of the image.

In one embodiment, as shown in FIG. 14, it is a schematic flow chart of synthesizing to obtain a full in-focus image in another embodiment. Synthesize the images corresponding to the phase difference of each target to obtain a fully in-focus image, including:

In operation 1402, the feature of the image corresponding to each target phase difference is extracted.

Specifically, FIG. 15 is a schematic flowchart of synthesizing to obtain a full in-focus image in another embodiment. The electronic device uses a convolutional neural network to perform convolution processing on the image corresponding to each target phase difference, and perform feature extraction. For example, in FIG. 15, convolution→feature extraction→convolution is performed on image 1. Perform convolution→feature extraction→convolution on image 2.

In operation 1404, the features of the image corresponding to each target phase difference are fused to obtain the first image feature.

Specifically, the electronic device fuses the features of the image corresponding to each target phase difference, and calculates the activation function to obtain the first image feature.

In operation 1406, the image corresponding to each target phase difference is averaged to obtain an average image.

Specifically, the electronic device performs averaging processing on the brightness value of the image corresponding to each target phase difference to obtain an average image.

Operation 1408: Perform feature extraction according to the average image and the first image feature to obtain the second image feature.

Specifically, the electronic device performs feature extraction according to the average image and the first image feature to obtain the second image feature.

Operation 1410: Perform feature reconstruction according to the second image feature and the average image to obtain a fully in-focus image.

Specifically, the electronic device performs feature reconstruction according to the second image feature and the average image to obtain a fully in-focus image.

The image processing method in the embodiment of the present application extracts the features of the image corresponding to each target phase difference, fuses the features of the image corresponding to each target phase difference, and obtains the first image feature. For the image corresponding to each target phase difference Perform averaging processing to obtain an average image, perform feature extraction based on the average image and the first image feature, obtain the second image feature, perform feature reconstruction based on the second image feature and the average image, and obtain a fully in-focus image. The neural network can be used. Synthesize images to improve the accuracy and clarity of image synthesis.

In one embodiment, acquiring the preview image includes: acquiring the region of interest in the preview image; and dividing the preview image into at least two subregions includes: dividing the region of interest in the preview image into at least two subregions.

Among them, the region of interest (ROI) refers to that in image processing, the area to be processed is outlined in the form of boxes, circles, ellipses, irregular polygons, etc., from the processed image. The area of interest can include background and objects. The electronic device receives the trigger instruction on the first preview image, and obtains the region of interest selected by the user according to the trigger instruction.

Specifically, the electronic device divides the region of interest into at least two sub-regions. The electronic device may divide the region of interest selected by the user into N×N sub-regions. Alternatively, the electronic device may divide the region of interest selected by the user into N×M sub-regions, etc., which is not limited thereto. Both N and M are positive integers.

The image processing method in the embodiment of the present application obtains the region of interest in the preview image, divides the region of interest in the preview image into at least two sub-regions, and can focus according to the region of interest, thereby ensuring the scene in the region of interest Clear, improve the image clarity of the region of interest in the full focus image.

In one embodiment, determining the at least two target phase differences according to the phase difference corresponding to each sub-region includes: acquiring a scene mode; and determining the at least two target phase differences according to the scene mode.

Specifically, each scene mode can correspond to different types of target phase differences. The scene mode may be a night scene mode, a panoramic mode, etc., and is not limited thereto. For example, the target phase difference corresponding to the A scene mode is a foreground phase difference and a background phase difference. The target phase difference corresponding to the B scene mode is the foreground phase difference, the median phase difference, and the background phase difference, etc., which are not limited to this.

The image processing method in the embodiment of the application obtains the scene mode, determines at least two target phase differences according to the scene mode, can quickly determine the target phase difference according to different scene modes, achieves the effect corresponding to different scenes, and improves image processing efficiency and image effects The clarity.

It should be understood that although the various operations in the flowcharts of FIGS. 2, 10, 13 and 14 are displayed in sequence as indicated by the arrows, these operations are not necessarily performed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these operations, and these operations can be executed in other orders. Moreover, at least part of the operations in Figures 2, 10, 13 and 14 may include multiple sub-operations or multiple stages. These sub-operations or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of the sub-operations or stages is not necessarily performed sequentially, but may be executed alternately or alternately with other operations or at least a part of the sub-operations or stages of other operations.

Fig. 16 is a structural block diagram of an image processing apparatus according to an embodiment. As shown in FIG. 16, an image processing device includes a preview image acquisition module 1602, a division module 1604, a phase difference acquisition module 1606, a focus module 1608, and a synthesis module 1610, in which:

The preview image acquisition module 1602 is used to acquire a preview image.

The dividing module 1604 is configured to divide the preview image into at least two sub-areas.

The phase difference acquiring module 1606 is configured to acquire the phase difference corresponding to each of the at least two sub-regions.

The phase difference acquisition module 1606 is further configured to determine at least two target phase differences from the phase differences corresponding to each sub-region, and the at least two target phase differences include the target foreground phase difference and the target background phase difference.

The focusing module 1608 is used for focusing according to the phase difference of each target to obtain an image corresponding to the phase difference of each target.

The synthesizing module 1610 is used to synthesize the images corresponding to each target phase difference to obtain a fully in-focus image.

The image processing device in this embodiment obtains a preview image, divides the preview image into at least two sub-regions, obtains the phase difference corresponding to each sub-region in the at least two sub-regions, and determines at least two phase differences from the phase difference corresponding to each sub-region. Target phase difference, at least two target phase differences include the target foreground phase difference and the target background phase difference. Focusing is performed according to the phase difference of each target to obtain an image corresponding to the phase difference of each target, and at least two targets at different focal points can be acquired. One is the background in-focus image, and the other is the foreground in-focus image. The images corresponding to the phase difference of each target are synthesized to obtain a fully in-focus image, which can obtain an image with less out-of-focus area and improve the image The clarity.

In one embodiment, the phase difference obtaining module 1606 is configured to divide the phase difference corresponding to each of the at least two sub-regions into a foreground phase difference set and a background phase difference set; and obtain the first phase difference mean value corresponding to the foreground phase difference set ; Obtain the second average phase difference corresponding to the background phase difference set; use the first average phase difference as the target foreground phase difference; use the second average phase difference as the target background phase difference.

The image processing device in this embodiment divides the phase difference corresponding to each of the at least two sub-regions into a foreground phase difference set and a background background phase difference set, and obtains the first phase difference mean value corresponding to the foreground phase difference set, and obtains The second average value of phase difference corresponding to the background phase difference set of the background, the first average value of phase difference is taken as the target foreground phase difference, and the second average value of phase difference is taken as the target background phase difference, the foreground in-focus image and background in-focus image can be obtained according to the average value Image, improve the clarity of the image.

In one embodiment, the phase difference acquisition module 1606 is used to exclude the maximum phase difference in the phase difference corresponding to the sub-region to obtain the remaining phase difference set; divide the remaining phase difference set into a foreground phase difference set and a background background phase difference set .

In the image processing device in this embodiment, since the furthest background details are often not important, the largest phase difference in the phase difference corresponding to the sub-region is excluded to obtain the remaining phase difference set, which can eliminate the farthest background and reduce the remaining phase difference. The set is divided into a foreground phase difference set and a background background phase difference set. Focusing based on the average value can improve the clarity of the image.

In one embodiment, the phase difference obtaining module 1606 is configured to obtain the maximum phase difference and the minimum phase difference among the phase differences of at least two sub-regions; the minimum phase difference is regarded as the foreground phase difference; and the maximum phase difference is regarded as the background phase difference.

The image processing device in this embodiment obtains the maximum phase difference and the minimum phase difference among the phase differences of at least two sub-regions, and uses the minimum phase difference as the foreground phase difference and the maximum phase difference as the background phase difference, so that only two The image is synthesized to improve image processing efficiency while improving image clarity.

In an embodiment, the electronic device includes an image sensor, the image sensor includes a plurality of pixel point groups arranged in an array, each of the pixel point groups includes M*N pixel points arranged in an array; each pixel point corresponds to one In the photosensitive unit, both M and N are natural numbers greater than or equal to 2. The phase difference obtaining module 1606 is used to obtain the target brightness map according to the brightness values of the pixels included in each pixel point group; perform segmentation processing on the target brightness map, and obtain the first segment brightness map and the second segment brightness map according to the results of the segmentation processing. Split brightness map; determine the phase difference of the matching pixels according to the position difference of the matching pixels in the first split brightness map and the second split brightness map; determine at least two sub-regions according to the phase difference of the matching pixels The phase difference corresponding to each sub-area in.

The image processing device in this embodiment obtains a target brightness map according to the brightness values of the pixel points included in each pixel point group in the image sensor, and after obtaining the target brightness map, performs segmentation processing on the target brightness map, according to As a result of the segmentation processing, the first segmented brightness map and the second segmented brightness map are obtained. Then, based on the position difference of the matching pixels in the first segmented brightness map and the second segmented brightness map, the pixels that match each other are determined Then, according to the phase difference of the matched pixels, the phase difference corresponding to each of the at least two sub-regions is determined. In this way, the brightness value of the pixel points included in each pixel point group in the image sensor can be used to determine the phase difference. Obtain the target phase difference map. Therefore, compared with the method of obtaining phase difference by using sparsely arranged phase detection pixels, the phase difference of the matched pixels in the embodiment of the present application contains relatively rich phase difference information. The accuracy of the phase difference obtained can be improved.

In one embodiment, the phase difference acquisition module 1606 is used to perform segmentation processing on the target brightness map to obtain multiple brightness map regions, each brightness map region includes a row of pixels in the target brightness map, or each brightness map region Including a column of pixels in the target brightness map; obtaining multiple first brightness map regions and multiple second brightness map regions from multiple brightness map regions, the first brightness map region including pixels in even rows of the target brightness map, or, The first luminance map area includes the pixels in the even-numbered columns of the target luminance map, the second luminance map area includes the pixels in the odd-numbered rows of the target luminance map, or the second luminance map area includes the pixels in the odd-numbered columns in the target luminance map; The first brightness map area composes a first segmented brightness map, and multiple second brightness map areas are used to compose a second segmented brightness map.

The image processing device in the embodiment of the present application does not need to shield the pixels to obtain the phase difference, and obtains relatively rich phase difference information by means of brightness segmentation, which improves the accuracy of the obtained phase difference.

In one embodiment, the phase difference acquisition module 1606 is configured to use the vertical phase difference as the phase difference corresponding to the sub-region when it is detected that the sub-region contains horizontal lines; when it is detected that the sub-region does not contain horizontal lines, the horizontal The phase difference is regarded as the phase difference corresponding to the sub-region.

In the image processing device in the embodiment of the present application, due to some characteristics of the camera, there may be a difference in imaging. That is, the lines may have problems such as smearing. Then when it is detected that the sub-region contains horizontal lines, the vertical phase difference is used as the phase difference corresponding to the sub-region; when it is detected that the sub-region contains vertical lines, the horizontal phase difference is used as the phase difference corresponding to the sub-region, which can improve the phase Poor acquisition accuracy, thereby improving image clarity.

In one embodiment, the focusing module 1608 is configured to use the sub-area corresponding to each target phase difference as a focus area to obtain an image corresponding to each target phase difference.

The image processing device in this embodiment uses the sub-area corresponding to each target phase difference as a focus area to obtain an image corresponding to each target phase difference, and can obtain images with different focal points for synthesis, thereby improving image clarity.

In one embodiment, the synthesis module 1610 is configured to divide the image corresponding to each target phase difference into the same number of sub-image areas; obtain the definition corresponding to each sub-image area; determine according to the definition corresponding to each sub-image area The sub-image area with the highest definition among the matching sub-image areas; the sub-image area with the highest definition is synthesized to obtain a fully in-focus image.

The image processing device in the embodiment of the present application divides the image corresponding to each target phase difference into the same number of sub-image areas; obtains the definition corresponding to each sub-image area; and determines the mutual resolution according to the corresponding definition of each sub-image area. The sub-image area with the highest definition among the matched sub-image areas; the sub-image area with the highest definition is synthesized to obtain a full in-focus image, which can quickly obtain a full-in-focus image and improve image processing efficiency.

In one embodiment, the synthesis module 1610 is used for convolution and sampling processing of the image corresponding to each target phase difference, and when the preset iterative conditions are met, the Gaussian pyramid of the image corresponding to each target phase difference is obtained; Each layer of the Gaussian pyramid of the image corresponding to the target phase difference is processed to obtain the Laplacian pyramid of the image corresponding to each target phase difference; the Laplacian pyramid of the image corresponding to each target phase difference is fused , The fused Laplacian pyramid is obtained; reconstruction processing is performed according to the fused Laplacian pyramid to obtain a fully in-focus image.

The image processing device in this embodiment performs convolution and sampling processing on the image corresponding to each target phase difference, and when the preset iterative conditions are met, the Gaussian pyramid of the image corresponding to each target phase difference is obtained, and the Gaussian pyramid is obtained according to each target phase difference. Each layer of the image in the Gaussian pyramid of the image corresponding to the difference is processed to obtain the Laplacian pyramid of the image corresponding to each target phase difference, and the Laplacian pyramid of the image corresponding to each target phase difference is merged to obtain The fused Laplacian Pyramid is reconstructed according to the fused Laplacian Pyramid to obtain a fully in-focus image, which can make the boundary between various regions more natural and improve the authenticity and clarity of the image.

In one embodiment, the synthesis module 1610 is used to extract the features of the image corresponding to each target phase difference; fuse the features of the image corresponding to each target phase difference to obtain the first image feature; The image is averaged to obtain an average image; feature extraction is performed based on the average image and the first image feature to obtain the second image feature; the second image feature is reconstructed based on the second image feature and the average image to obtain a fully in-focus image.

The image processing device in the embodiment of the present application extracts the features of the image corresponding to each target phase difference, fuses the features of the image corresponding to each target phase difference, and obtains the first image feature. For the image corresponding to each target phase difference Perform averaging processing to obtain an average image, perform feature extraction based on the average image and the first image feature, obtain the second image feature, perform feature reconstruction based on the second image feature and the average image, and obtain a fully in-focus image. The neural network can be used Synthesize images to improve the accuracy and clarity of image synthesis.

In one embodiment, the preview image acquisition module 1602 is used to acquire the region of interest in the preview image. The dividing module 1604 is configured to divide the region of interest in the preview image into at least two sub-regions.

The image processing device in the application embodiment obtains the region of interest in the preview image, divides the region of interest in the preview image into at least two sub-regions, and can focus according to the region of interest, thereby ensuring that the scene in the region of interest is clear , To improve the image clarity of the region of interest in the all-in-focus image.

In one embodiment, the phase difference obtaining module 1606 is used to obtain a scene mode; and determine at least two target phase differences according to the scene mode.

The image processing device in the embodiment of the present application obtains the scene mode, determines at least two target phase differences according to the scene mode, can quickly determine the target phase difference according to different scene modes, achieves the effect corresponding to different scenes, and improves image processing efficiency and image effects The clarity.

The division of the modules in the above-mentioned image processing apparatus is only for illustration. In other embodiments, the image processing apparatus may be divided into different modules as required to complete all or part of the functions of the above-mentioned image processing apparatus.

For the specific definition of the image processing device, please refer to the above definition of the image processing method, which will not be repeated here. Each module in the above-mentioned image processing device may be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

Fig. 17 is a schematic diagram of the internal structure of an electronic device in an embodiment. As shown in FIG. 17, the electronic device includes a processor and a memory connected through a system bus. Among them, the processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory may include a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image processing method provided in the following embodiments. The internal memory provides a cached operating environment for the operating system computer program in the non-volatile storage medium. The electronic device can be a mobile phone, a tablet computer, or a personal digital assistant or a wearable device.

The implementation of each module in the image processing apparatus provided in the embodiment of the present application may be in the form of a computer program. The computer program can be run on a terminal or a server. The program module composed of the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, the operation of the method described in the embodiment of the present application is realized.

The embodiment of the present application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to perform the operations of the image processing method.

A computer program product containing instructions that, when run on a computer, causes the computer to execute an image processing method.

Any reference to memory, storage, database, or other media used in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which acts as external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The above-mentioned embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the protection scope of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims

An image processing method, characterized in that it is applied to electronic equipment, and includes:

Obtain a preview image;

Dividing the preview image into at least two sub-areas;

Acquiring a phase difference corresponding to each of the at least two sub-regions;

Determining at least two target phase differences from the phase differences corresponding to each sub-region, where the at least two target phase differences include a target foreground phase difference and a target background phase difference;

Performing focusing according to the phase difference of each target to obtain an image corresponding to the phase difference of each target; and

Synthesis is performed according to the images corresponding to the phase difference of each target to obtain a fully in-focus image.
The method according to claim 1, wherein the at least two target phase differences are determined from the phase differences corresponding to each sub-region, and the at least two target phase differences include the target foreground phase difference and the target phase difference. Background phase difference, including:

Dividing the phase difference corresponding to each of the at least two sub-regions into a foreground phase difference set and a background phase difference set;

Acquiring a first mean value of phase difference corresponding to the foreground phase difference set;

Acquiring a second mean value of phase difference corresponding to the background phase difference set;

Use the first mean value of the phase difference as the target foreground phase difference;

The second mean value of the phase difference is used as the target background phase difference.
The method according to claim 2, wherein the method further comprises:

Excluding the largest phase difference among the phase differences corresponding to the sub-regions to obtain a residual phase difference set;

The dividing the phase difference corresponding to each of the at least two sub-regions into a foreground phase difference set and a background phase difference set includes:

The remaining phase difference set is divided into a foreground phase difference set and a background phase difference set.
The method according to claim 1, wherein the at least two target phase differences are determined from the phase differences corresponding to each sub-region, and the at least two target phase differences include a foreground phase difference and a background phase Poor, including:

Obtaining the maximum phase difference and the minimum phase difference among the phase differences of the at least two sub-regions;

Taking the minimum phase difference as the foreground phase difference;

The maximum phase difference is regarded as the background phase difference.
The method according to claim 1, wherein the electronic device comprises an image sensor, the image sensor comprises a plurality of pixel point groups arranged in an array, and each of the pixel point groups comprises M* arranged in an array. N pixels; each pixel corresponds to a photosensitive unit, where M and N are both natural numbers greater than or equal to 2;

The acquiring the phase difference corresponding to each of the at least two sub-regions includes:

Acquiring a target brightness map according to the brightness value of the pixel points included in each pixel point group;

Performing segmentation processing on the target brightness map, and obtaining a first segmented brightness map and a second segmented brightness map according to the results of segmentation processing;

Determine the phase difference of the pixels that match each other according to the position difference of the pixels that match each other in the first split brightness map and the second split brightness map;

The phase difference corresponding to each of the at least two sub-regions is determined according to the phase difference of the mutually matched pixels.
The method according to claim 5, wherein the segmentation processing on the target brightness map, and obtaining the first segmented brightness map and the second segmented brightness map according to the results of the segmentation processing, comprises:

The target brightness map is segmented to obtain multiple brightness map regions, each of the brightness map regions includes a row of pixels in the target brightness map, or each of the brightness map regions includes the target brightness A column of pixels in the picture;

Acquire a plurality of first brightness map regions and a plurality of second brightness map regions from the plurality of brightness map regions, where the first brightness map region includes pixels in even rows of the target brightness map, or the first brightness map region A luminance map area includes pixels in even-numbered columns in the target luminance map, the second luminance map area includes pixels in odd rows in the target luminance map, or the second luminance map area includes the target luminance map Pixels in odd columns;

The plurality of first brightness map regions are used to form the first segmented brightness map, and the plurality of second brightness map regions are used to form the second segmented brightness map.
The method according to claim 1, wherein the electronic device comprises an image sensor, the image sensor comprises a plurality of pixel point groups arranged in an array, and each of the pixel point groups comprises M* arranged in an array. N pixel points; each pixel point corresponds to a photosensitive unit, where M and N are both natural numbers greater than or equal to 2; the phase difference corresponding to each sub-region includes a horizontal phase difference and a vertical phase difference;

The acquiring the phase difference corresponding to each of the at least two sub-regions includes:

When it is detected that the sub-region contains horizontal lines, use the vertical phase difference as the phase difference corresponding to the sub-region;

When it is detected that the sub-region does not contain horizontal lines, the horizontal phase difference is taken as the phase difference corresponding to the sub-region.
The method according to claim 1, wherein the focusing according to the phase difference of each target to obtain an image corresponding to the phase difference of each target comprises:

The sub-region corresponding to each target phase difference is used as a focus area, and an image corresponding to each target phase difference is obtained.
The method according to any one of claims 1 to 8, wherein the synthesizing images corresponding to the phase difference of each target to obtain a fully in-focus image comprises:

Dividing the image corresponding to each target phase difference into the same number of sub-image areas;

Acquiring the definition corresponding to each sub-image area;

Determining the sub-image area with the highest definition among the sub-image areas that match each other according to the corresponding definition of each sub-image area;

The sub-image areas with the highest definition are spliced and synthesized to obtain a fully in-focus image.
The method according to any one of claims 1 to 8, wherein the synthesizing images corresponding to the phase difference of each target to obtain a fully in-focus image comprises:

For the image convolution and sampling processing corresponding to each target phase difference, when a preset iterative condition is met, a Gaussian pyramid of the image corresponding to each target phase difference is obtained;

Processing according to each layer of the image in the Gaussian pyramid of the image corresponding to each target phase difference to obtain the Laplacian pyramid of the image corresponding to each target phase difference;

Fusing the Laplacian pyramid of the image corresponding to the phase difference of each target to obtain a fused Laplacian pyramid;

Perform reconstruction processing according to the fused Laplacian pyramid to obtain a fully in-focus image.
The method according to any one of claims 1 to 8, wherein the synthesizing images corresponding to the phase difference of each target to obtain a fully in-focus image comprises:

Extracting the feature of the image corresponding to each target phase difference;

Fusing the features of the image corresponding to the phase difference of each target to obtain the first image feature;

Performing averaging processing on the image corresponding to each target phase difference to obtain an average image;

Performing feature extraction according to the average image and the first image feature to obtain a second image feature;

Perform feature reconstruction according to the second image feature and the average image to obtain a fully in-focus image.
The method according to any one of claims 1 to 8, wherein said obtaining a preview image comprises:

Obtain the region of interest in the preview image;

The dividing the preview image into at least two sub-areas includes:

The region of interest in the preview image is divided into at least two sub-regions.
An image processing device, characterized in that it comprises:

Preview image acquisition module for acquiring preview images;

A dividing module, configured to divide the preview image into at least two sub-areas;

A phase difference acquiring module, configured to acquire the phase difference corresponding to each of the at least two sub-regions;

The phase difference acquisition module is further configured to determine at least two target phase differences from the phase difference corresponding to each sub-region, and the at least two target phase differences include a target foreground phase difference and a target background phase difference;

A focusing module, configured to perform focusing according to the phase difference of each target to obtain an image corresponding to the phase difference of each target;

The synthesis module is used for synthesizing the image corresponding to each target phase difference to obtain a fully in-focus image.
The apparatus according to claim 13, wherein the phase difference obtaining module is further configured to divide the phase difference corresponding to each of the at least two sub-regions into a foreground phase difference set and a background phase difference set;

Acquiring a first mean value of phase difference corresponding to the foreground phase difference set;

Acquiring a second mean value of phase difference corresponding to the background phase difference set;

Use the first mean value of the phase difference as the target foreground phase difference;

The second mean value of the phase difference is used as the target background phase difference.
The apparatus according to claim 14, wherein the phase difference acquisition module is configured to exclude the maximum phase difference among the phase differences corresponding to the sub-regions to obtain a residual phase difference set; and divide the residual phase difference set into foreground The phase difference set and the background phase difference set.
The device according to claim 13, wherein the phase difference obtaining module is configured to obtain the maximum phase difference and the minimum phase difference among the phase differences of at least two sub-regions;

Taking the minimum phase difference as the foreground phase difference;

The maximum phase difference is regarded as the background phase difference.
The device according to claim 13, wherein the electronic device comprises an image sensor, the image sensor comprises a plurality of pixel point groups arranged in an array, and each of the pixel point groups comprises M*N pixel point groups arranged in an array. Pixels; each pixel corresponds to a photosensitive unit, where M and N are both natural numbers greater than or equal to 2;

The phase difference obtaining module is configured to obtain a target brightness map according to the brightness value of the pixel points included in each pixel point group;

Performing segmentation processing on the target brightness map, and obtaining a first segmented brightness map and a second segmented brightness map according to the results of segmentation processing;

Determine the phase difference of the pixels that match each other according to the position difference of the pixels that match each other in the first split brightness map and the second split brightness map;

The phase difference corresponding to each of the at least two sub-regions is determined according to the phase difference of the mutually matched pixels.
The device according to claim 17, wherein the phase difference acquisition module is used to perform segmentation processing on the target brightness map to obtain a plurality of brightness map regions, each of the brightness map regions including the target A row of pixels in the brightness map, or each of the brightness map regions includes a column of pixels in the target brightness map;

Acquire a plurality of first brightness map regions and a plurality of second brightness map regions from the plurality of brightness map regions, where the first brightness map region includes pixels in even rows of the target brightness map, or the first brightness map region A luminance map area includes pixels in even-numbered columns in the target luminance map, the second luminance map area includes pixels in odd rows in the target luminance map, or the second luminance map area includes the target luminance map Pixels in odd columns;

The multiple first brightness map regions are used to form the first split brightness map, and the multiple second brightness map regions are used to form the second split brightness map.
An electronic device, comprising a memory and a processor, and a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes any one of claims 1 to 12 The steps of the image processing method.
A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method according to any one of claims 1 to 12 when the computer program is executed by a processor.