CN116797447A

CN116797447A - Image processing method and device, electronic equipment and storage medium

Info

Publication number: CN116797447A
Application number: CN202210266377.6A
Authority: CN
Inventors: 陈沭
Original assignee: Beijing Xiaomi Mobile Software Co Ltd
Current assignee: Beijing Xiaomi Mobile Software Co Ltd
Priority date: 2022-03-17
Filing date: 2022-03-17
Publication date: 2023-09-22

Abstract

The disclosure relates to an image processing method and device, electronic equipment and storage medium. Wherein the method comprises the following steps: acquiring an image to be processed and depth information thereof; performing background separation on the image to be processed based on the depth information to obtain a foreground image containing the shot subject and a background image not containing the shot subject; and carrying out fusion processing on the foreground image and the background image so that part of the shot main body in the fused image is not overlapped with the background image.

Description

Image processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of image processing, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

With the advent of the national photography era, various manufacturers have carried various photography functions for electronic products in order to improve the attractiveness of their own products, so as to achieve corresponding photography effects.

In the related art, in order to achieve the effect of highlighting a subject in an image, background recognition is generally performed on the image, and the background obtained by the recognition is subjected to blurring, thereby achieving the effect of highlighting the subject.

Disclosure of Invention

The present disclosure provides an image processing method and apparatus, an electronic device, and a storage medium, capable of highlighting a subject in an image.

According to a first aspect of the present disclosure, there is provided an image processing method including:

acquiring an image to be processed and depth information thereof;

performing background separation on the image to be processed based on the depth information to obtain a foreground image containing the shot subject and a background image not containing the shot subject;

and carrying out fusion processing on the foreground image and the background image so that part of the shot main body in the fused image is not overlapped with the background image.

According to a second aspect of the present disclosure, there is provided an image processing apparatus including:

the acquisition unit acquires an image to be processed and depth information of the image;

a separation unit for performing background separation on the image to be processed based on the depth information to obtain a foreground image containing the shot main body and a background image not containing the shot main body;

and the fusion unit is used for carrying out fusion processing on the foreground image and the background image so that part of the shot main body in the fused image is not overlapped with the background image.

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor implements the method of the first aspect by executing the executable instructions.

According to a fourth aspect of the present disclosure there is provided a computer readable storage medium having stored thereon computer instructions which when executed by a processor perform the steps of the method according to the first aspect.

In the technical scheme of the disclosure, the depth information of the image to be processed can be utilized to perform background separation on the image to be processed so as to obtain a foreground image containing the shot subject and a background image not containing the shot subject. On the basis, the two can be further fused. The fusion treatment standard is as follows: in the fused image, a part of the subject is made to overlap with the background image.

It should be understood that since the subject has a portion that does not overlap with the background image in the fused image, an effect similar to the subject extending from the image to the outside of the image can be created, highlighting the subject to a greater extent.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a flow chart of an image processing method shown in an exemplary embodiment of the present disclosure;

FIG. 2 is a flowchart of another image processing method shown in an exemplary embodiment of the present disclosure;

FIG. 3 is a schematic illustration of an image to be processed shown in accordance with an exemplary embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a foreground image and a background image shown in an exemplary embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a position adjustment process shown in an exemplary embodiment of the present disclosure;

FIG. 6 is a schematic illustration of a fused image shown in an exemplary embodiment of the present disclosure;

FIG. 7 is a flowchart of another image processing method shown in an exemplary embodiment of the present disclosure;

FIG. 8 is a schematic diagram of a background image cropping process, as shown in an exemplary embodiment of the present disclosure;

FIG. 9 is a schematic illustration of another fused image shown in an exemplary embodiment of the present disclosure;

fig. 10 is a block diagram of an image processing apparatus shown in an exemplary embodiment of the present disclosure;

FIG. 11 is a block diagram of another image processing apparatus shown in an exemplary embodiment of the present disclosure;

fig. 12 is a schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.

In the related art, in order to highlight a subject in an image, depth information of the image is generally acquired while image capturing is performed, and blurring processing is performed on picture content belonging to a background in the image according to the acquired depth information, so as to achieve the purpose of highlighting the subject.

However, although this approach can highlight the subject to some extent, in practical applications, the highlighting effect is not obvious. In many scenes, the user cannot lock the subject from the image at a glance.

For this reason, the present disclosure proposes an image processing method to avoid the problem in the related art that the salient effect of the subject in the image is not obvious due to blurring only the picture belonging to the background.

Fig. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. As shown in fig. 1, the method may include the steps of:

step 102, obtaining an image to be processed and depth information thereof.

As is clear from the above description, the related art realizes the prominent effect of the subject by blurring the picture content belonging to the background. However, the salient effect of this approach is not obvious. In view of this, the present disclosure does not adopt a manner of blurring the picture content belonging to the background in the image to be processed any more, so as to achieve the highlighting of the subject.

In the present disclosure, an image to be processed and depth information of the image to be processed may be preferentially acquired, and background separation is performed on the image to be processed based on the acquired depth information, so as to obtain a foreground image including a subject and a background image not including the subject. On the basis, fusion processing can be further carried out on the foreground image and the background image. The fusion treatment standard is as follows: in the fused image, a part of the subject does not overlap with the background image.

It should be understood that, in the image obtained through the fusion process, a part of the subject does not overlap with the background image, which corresponds to a part of the subject being located inside the background image and another part being located outside the background image, and therefore, an effect that the subject extends from the background contained in the image to outside the image can be created. In this scene, when the user views the image, a subject in the image can generally be captured directly, not a background screen. Obviously, the subject can be more effectively highlighted than the subject highlighting method in the related art.

It should be stated that the size of the foreground image and the size of the background image obtained by the background separation of the present disclosure generally coincide with the size of the image to be processed. The foreground image only contains the picture content of the shot main body, and other areas except the area where the shot main body is positioned are blank pictures; the background image only includes background pictures except the shot main body, and the area where the shot main body is positioned is a blank area. Of course, this case is merely illustrative, and in other cases, the sizes of the foreground image and the background image may be different from the image to be processed, and the sizes thereof may be determined by those skilled in the art according to actual situations, which is not limited by the present disclosure.

In the present disclosure, a depth camera may be started to acquire depth information of an image to be processed while image capturing is performed. In some possible embodiments, the electronic device may call an RGB camera and a ToF (Time of flight) camera to capture a subject at the same Time when detecting an image capturing instruction. The RGB image is used for image acquisition to acquire an image to be processed; and the ToF camera is used to obtain depth information for each position in the image to be processed. The shooting mode is double shooting in the related art, and can correspond each piece of depth information acquired by the depth camera to each position in the image to be processed, so that the situation that the background separation is inaccurate due to the fact that the depth information of each position in the image to be processed is determined in an error mode is avoided.

In the present disclosure, a detection signal transmitting assembly may be assembled in an electronic device as the above-described depth camera. Then, after the electronic equipment detects an image shooting instruction initiated by a user, on one hand, the electronic equipment can call an image sensor to acquire an image so as to obtain an image to be processed; on the other hand, a ranging signal may be transmitted in a direction prescribed by the subject to acquire depth information of an image to be processed. Wherein the ranging signal returns after passing through the collision to be received by the electronic device. On the basis, the electronic equipment can determine a time difference based on the time when the ranging signal is sent and the time when the ranging signal is received, so that depth information corresponding to each position in the image to be processed is determined according to the time difference. It should be understood that the above time difference is the propagation duration of the ranging signal, and the propagation speed of the ranging signal is known, and obviously, after the propagation duration is obtained, the distance between the electronic device and the real object corresponding to each position in the photographed image to be processed can be obtained as the depth information of the corresponding position.

Of course, the above manner of determining the depth information of each position in the image to be processed is merely illustrative, and how to obtain the depth information of each position in the image to be processed can be set by those skilled in the art according to the actual situation, which is not limited in this disclosure.

And 104, carrying out background separation on the image to be processed based on the depth information to obtain a foreground image containing the shot subject and a background image not containing the shot subject.

In the present disclosure, after an image to be processed and depth information thereof are obtained, background separation of the image to be processed may be performed based on the depth information. In some possible embodiments, the proportion of each depth information in the image to be processed may be preferentially determined, so as to further determine the depth range of the position of the subject according to the determined proportion. On this basis, the screen content whose depth information is within the depth range can be determined as the screen content included in the foreground image, and the other screen can be determined as the screen content included in the background image.

The depth range of the position of the subject can be determined by plotting a curve. For example, after obtaining depth information of each position in an image to be processed, the present disclosure may draw a depth information distribution curve for representing the proportion of each depth information in the image to be processed, on this basis, two depth ranges with the highest duty ratio may be determined from the curve, and the depth range with the smaller represented distance may be determined as the depth range of the position where the subject is located.

It should be appreciated that during actual photographing, the user is generally photographing the subject, and the object constituting the background is located behind the subject to better highlight the subject in the image. In the scene, the proportion of the shot main body and the background object in the image is large, and the distance represented by the depth information of the position of the shot main body is smaller than the distance represented by the depth information of the position of the background picture. Therefore, among the plurality of depth ranges, the depth range with the smaller represented distance is the depth range of the position of the photographed subject. In this way, the depth range of the position of the subject can be accurately determined.

And 106, carrying out fusion processing on the foreground image and the background image so that part of the shot object in the fused image is not overlapped with the background image.

In the present disclosure, after a foreground image and a background image are obtained by background separation, the foreground image and the background image may be fused, where the fusion method used must be such that a part of the image obtained by fusion does not overlap with the background image.

In an embodiment, the foreground image and the background image obtained by separating the background can be directly subjected to image fusion without any additional processing. For example, before fusion, the position of at least one of the foreground image and the background image may be preferentially adjusted so that the foreground image and the background image are dislocated, and then the foreground image and the background image are dislocated and fused. Of course, in order to ensure that there is a non-overlapping portion of the subject and the background image in the fused image, the distance of the misalignment should be sufficient to allow the subject to go beyond the range covered by the background image, but not completely beyond the range. In actual operation, the adjustment of the foreground image or the adjustment of the background image, and the adjustment direction and distance can be set by those skilled in the art according to actual requirements. Of course, besides being set by those skilled in the art according to requirements, the foreground image and the background image may also be displayed in an image capturing application, so that a user may manually adjust the background image or the background image, and particularly how the foreground image and the background image are dislocated may be determined according to practical situations, which is not limited by the present disclosure.

In another embodiment, at least one of the foreground image and the background image obtained by separating the background may be preferentially adjusted, and after the adjustment is completed, the two images are fused.

In one case of the present embodiment, the background image may be subjected to reduction processing such that at least a part of the subject included in the foreground image cannot overlap with the reduced background image. Wherein the scaling down may be determined in a number of ways. For example, it may be determined by the area occupied by the subject in the foreground image; or can be selected from a plurality of preset proportions to be selected.

In another case of the present embodiment, the foreground image may be subjected to the enlargement processing such that at least a part of the subject included in the enlarged foreground image cannot overlap with the background image. The scale of the magnification may be determined in various ways, for example, by the area occupied by the subject in the foreground image, or may be selected from a plurality of preset proportions to be selected, similar to the above case.

In still another case of the present embodiment, the background image may be subjected to clipping processing such that at least a part of the subject included in the foreground image cannot overlap with the clipped background image. The clipping proportion can also be determined in various manners, for example, can be determined by the occupied area of the shot main body in the foreground image, and can also be selected from a plurality of preset proportions to be selected.

In this case, a clipping template may be generated based on the determined clipping ratio, and a coverage area of the clipping template and a subject included in the foreground image may have a non-overlapping portion, so as to ensure that a non-overlapping region exists between the background image and the subject after clipping. On the basis, the background image can be cut based on the cutting template. The cutting template and the background image can be further displayed to the user, so that the user can control the cutting effect by manually adjusting the relative positions of the cutting template and the background image, and the final highlighting effect is indirectly influenced.

It should be noted that, in addition to the adjustment operation before image fusion in any of the above three cases, the adjustment operation before fusion may be performed in at least two of the above three cases, specifically, several adjustment operations may be adopted, and which adjustment operation may be determined by those skilled in the art according to the actual situation, which is not limited in this disclosure. Of course, the criteria for determining the proportion are consistent regardless of which or which of the above adjustment methods is adopted, so that a part of the subject is not overlapped with the background image, and how to determine the proportion specifically can be determined by those skilled in the art according to the actual situation, which is not limited by the present disclosure.

In the present disclosure, before image fusion is performed, a foreground image and a background image may also be displayed to a user, so that the user can adjust the relative positional relationship between the foreground image and the background image by himself. On the basis, the method and the device can determine the overlapped part of the foreground image and the background image as the area to be fused according to the adjustment result of the user, and further perform image fusion on the partial area. It should be noted that the foreground image and the background image adjusted by the user may be a foreground image and a background image obtained by separating a background, or may be a foreground image and a background image obtained by the above adjustment operation, and specifically whether the foreground image and the background image have been adjusted or not may be determined by a person skilled in the art according to the actual situation, which is not limited in this disclosure.

In the present disclosure, since the region in the background image that is originally covered by the subject is a blank region, and the region involved in the image fusion operation may not include the blank region, after the fusion operation is completed, a completion operation may also be performed on the blank region in the fused image. For example, the completion of the blank area may be achieved by copying another background screen to the blank area. Of course, this example is merely illustrative, and it is specifically whether or not the image complement operation is required and how to perform the image complement operation may be determined by those skilled in the art according to actual circumstances, which is not limited by the present disclosure.

In the present disclosure, an image frame may also be provided for decorating the fused image. Wherein, the lamination relation of the shot main body and the image frame is: the subject is located above the image frame (i.e., in the overlapping region of the image frame and the subject, the subject covers the image frame). In some possible embodiments, the inner dimension of the image frame may be consistent with the background image, and the outer dimension may be greater than the background image, so that in the fused image, a portion of the subject that does not overlap with the background image may be covered on the image frame. It should be understood that, setting the image frame is equivalent to manufacturing a visual buffer area between the background image and the live-action outside the image, so that the problem that the whole display effect of the image and the salient effect of the shot main body cannot be considered due to too abrupt of the shot main body on the basis of realizing the effect that the shot main body extends from the background of the image to outside the background is avoided.

It should also be stated that the technical solution of the present disclosure may be applied to any type of electronic device, for example, the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or a fixed terminal such as a smart television, a PC (personal computer ), etc. It should be understood that, only electronic devices having an image processing function may be used as an execution body of the present disclosure, and in particular, which type of electronic device is used as an execution body of the technical solution of the present disclosure may be determined by those skilled in the art according to actual needs, which is not limited by the present disclosure.

As can be seen from the above description, after the image to be processed and the depth information thereof are obtained, the disclosure may perform background separation on the image to be processed based on the depth information, so as to obtain a foreground image including the subject and a background image not including the subject. On the basis, the foreground image and the background image can be subjected to image fusion, wherein the fusion standard is as follows: so that a part of the subject in the fused image does not overlap with the background image.

It should be understood that a part of the subject does not overlap with the background image, which is equivalent to that a part of the subject is located within the coverage of the background image and a part of the subject is located outside the coverage of the background image, so that the effect that the subject extends from the background image to the outside of the image can be created.

Furthermore, the present disclosure may further adjust at least one of the background image and the foreground image obtained by separating the background before performing image fusion, and perform image fusion after the adjustment. For example, the background image may be cut preferentially, and then the foreground image and the cut background image may be fused. It is to be understood that, after the adjustment, even if the position adjustment is not performed on the foreground image and the background image, a part of the subject in the fused image can be made to overlap with the background image.

Still further, the present disclosure may further provide an image frame for the fused image, where a portion of the subject that does not overlap with the background image covers over the image frame. It is easy to see that the image frame is equivalent to a visual buffer area between the background picture and the live-action outside the background picture, so that the problem that the overall display effect of the image and the salient effect of the shot main body cannot be considered due to the fact that the shot main body is too abrupt is avoided.

In the following, taking double photographing by a smart phone as an example, the technical scheme of the disclosure is introduced.

Fig. 2 is a flowchart illustrating another image processing method according to an exemplary embodiment of the present disclosure. As shown in fig. 2, the method comprises the steps of:

in step 201, an image capturing instruction is received.

In this embodiment, the smart phone may have a camera application installed therein in advance. Wherein a body protrusion mode may be included in the camera application. Then, the user can select the main body protruding mode to instruct the smart phone to process the image after the smart phone shoots the image, so as to highlight the position of the shot main body in the image.

Step 202, calling an RGB camera to shoot an image.

In the present embodiment, an RGB camera may be mounted for acquiring a to-be-processed image including a subject.

For example, the captured image to be processed may be as shown in fig. 3. The subject is a flower 31, and the background screen is a lawn 32 (for convenience of representation, the area covered by the lawn is represented by a scattered filling manner in the figure).

Step 203, calling a depth camera to acquire depth information.

In this embodiment, while image capturing is performed, a depth camera may be called to acquire depth information corresponding to an image to be processed. The depth camera may include a ranging signal transmitting component and a ranging signal receiving component.

With the above example in mind, the ranging signal transmitting assembly can transmit ranging signals to the flowers 31 and the grasslands 32 behind them, and then the ranging signals are returned after hitting the flowers 31 or the grasslands 32 and detected by the ranging signal receiving assembly. On the basis, the smart phone can calculate and obtain the distance corresponding to each position in the image to be processed based on the sending time and the receiving time of each ranging signal, and the distance is used as the depth information of the corresponding position.

In step 204, background separation is performed on the captured image based on the depth information.

In this embodiment, after the image to be processed and the depth information thereof are acquired, the background separation of the image to be processed may be performed based on the depth information.

For example, the distance between the smart phone and each position of the flower 31 is within a relatively similar distance range, and the distance between the smart phone and each position of the lawn 32 is also within a relatively similar distance range. Thus, at least two regions composed of respective positions close in distance can be preferentially determined. Since the distance between the flower 31 and the smart phone is obviously smaller than the distance between the lawn 32 and the smart phone, the area with a smaller distance range value may be determined as the area where the flower 31 is located, and the area with a relatively larger distance range value may be determined as the area where the lawn 32 is located. On this basis, a foreground image 41 including flowers 31 as shown in fig. 4 and a background image 42 including grasslands 32 can be obtained.

Step 205, displaying the separated foreground image and background image to the user.

In this embodiment, after the foreground image and the background image are obtained through background separation, the foreground image and the background image can be displayed in a display interface of the camera application, so that a user can adjust the relative positional relationship between the foreground image and the background image.

Step 206, receiving a position adjustment instruction of the user.

With the above example in mind, as shown in fig. 5, the user may continuously press the foreground image or the background image to move the selected image, thereby determining the region where background fusion is required.

Step 207, adjusting the relative positional relationship between the foreground image and the background image.

And step 208, performing image fusion based on the adjusted relative position relationship.

Following the above example, after background fusion is completed, an image as shown in fig. 6 can be obtained. Of course, it should be emphasized that, in the present embodiment, since the flower 31 in the foreground image 41 cannot block the blank area in the background image 42, it is also necessary to perform screen complement on the blank area in the background image 42. The specific complement means may be determined by those skilled in the art according to actual requirements, and the present embodiment is not limited thereto.

According to the technical scheme, the smart phone in the embodiment can obtain the image to be processed through shooting by the RGB camera, and obtain the depth information of the image to be processed by the depth camera, so that the background separation aiming at the image to be processed is realized. On the basis, the effect that the shot main body extends from the background picture to the outside of the background picture can be realized by carrying out dislocation fusion on the background image and the foreground image which are obtained by separating the background, and the problem that the salient effect of the shot main body is not obvious in the related art is avoided.

In the following, another embodiment of the technical solution of the present disclosure is described by taking a smart phone as an example. It should be noted that, in the next embodiment, the operation manners of steps 301 to 304 are similar to those of the previous embodiment, and the related content may refer to the description of the previous embodiment, which will not be described in detail below.

Fig. 7 is a flowchart of yet another image processing method shown in an exemplary embodiment of the present disclosure. As shown in fig. 7, the method includes the steps of:

in step 701, an image capturing instruction is received.

Step 702, call an RGB camera to take an image.

In step 703, the depth camera is invoked to obtain depth information.

In step 704, background separation is performed on the captured image based on the depth information.

Step 705, clipping the background image.

Unlike the previous embodiment, the present embodiment may further clip the background image after the foreground image and the background image are obtained by the background separation, so that the subject in the foreground image and the clipped background image have a part that does not overlap.

For example, after obtaining the foreground image 41 and the background image 42 shown in fig. 4, the cropping process may be as shown in fig. 8. In practical applications, the background image 42, and a clipping template (also referred to as a clipping mask) shown by a dotted line may also be presented to the user, so that the user adjusts the clipping range by himself.

And step 706, performing image fusion on the foreground image and the cut background image according to the original relative position relation.

Taking the above example, after the clipping to obtain the clipped background image 43 shown in fig. 8, the foreground image 41 and the clipped background image 43 are subjected to image fusion according to the original relative positional relationship (i.e., the relative positional relationship in which the edges overlap each other without position adjustment, i.e., as shown in the image a in fig. 9) of the foreground image 41 and the background image 42, so as to obtain the image B shown in fig. 9. In practical applications, an image frame may be added to the fused image, which may also be referred to as a photo frame in a photo, for example, the image with the added image frame may be shown as image C in fig. 9.

As can be seen from the above technical solution, in this embodiment, compared with the previous embodiment, the clipping operation is preferentially performed on the background image, so that there is a non-overlapping portion between the clipped background image and the subject included in the foreground image. On the basis, the effect that the shot main body extends from the background picture to the outside of the background picture can be realized by carrying out image fusion on the foreground image and the cut background image, and compared with the mode of blurring only the background in the related art, the shot main body can be highlighted to a greater extent.

Fig. 10 is a block diagram of an image processing apparatus shown in an exemplary embodiment of the present disclosure. Referring to fig. 10, the apparatus includes an acquisition unit 1001, a separation unit 1002, and a fusion unit 1003.

An acquisition unit 1001 that acquires an image to be processed and depth information thereof;

a separation unit 1002, configured to perform background separation on the image to be processed based on the depth information, to obtain a foreground image including the subject and a background image not including the subject;

and a fusion unit 1003 that performs fusion processing on the foreground image and the background image so that a part of the subject in the fused image does not overlap with the background image.

Optionally, the obtaining unit 1001 is further configured to:

responding to an image shooting instruction initiated by a user, calling an image sensor to execute image acquisition operation so as to acquire an image to be processed, and transmitting a ranging signal to the direction prescribed by the shot main body;

and receiving the ranging signal returned after collision, and determining depth information corresponding to each position in the image to be processed based on the time difference between sending the ranging signal and receiving the ranging signal.

Optionally, the separation unit 1002 is further configured to:

Determining the depth range of the position of the shot main body based on the proportion of each depth information in the image to be processed;

determining picture contents of which depth information is within the depth range as picture contents contained in the foreground image, and determining other picture contents as picture contents contained in the background image.

Optionally, the separation unit 1002 is further configured to:

drawing a depth information distribution curve for representing the proportion of each piece of depth information in the image to be processed based on the depth information of each position in the image to be processed;

and determining at least two depth ranges with the highest duty ratio from the depth information distribution curve, and determining the depth range with the smaller represented distance as the depth range of the position of the shot main body.

Optionally, the fusion unit 1003 is further configured to:

and carrying out dislocation fusion on the foreground image and the background image so that part of the shot main body contained in the foreground image is not overlapped with the background image in the fused image.

Optionally, the fusion unit 1003 is further configured to:

and adjusting at least one of the foreground image and the background image, and fusing the foreground image and the background image after the adjustment is completed, so that part of the shot main body in the fused image is not overlapped with the background image.

Optionally, the fusion unit 1003 is further configured to:

performing reduction processing on the background image; and/or the number of the groups of groups,

clipping the background image; and/or the number of the groups of groups,

and amplifying the foreground image.

Optionally, the fusion unit 1003 is further configured to:

determining a clipping template based on the foreground image, wherein the coverage area of the clipping template has a non-overlapping part with a shot subject in the foreground image;

and clipping the background image based on the clipping template.

As shown in fig. 11, fig. 11 is a block diagram of another image processing apparatus according to an exemplary embodiment of the present disclosure, which further includes, on the basis of the foregoing embodiment shown in fig. 10: a determining unit 1004.

Optionally, the method further comprises:

and a determining unit 1004 for determining a middle-view area included in the background image based on the depth information, and performing blurring processing on a picture belonging to the middle-view area.

For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the objectives of the disclosed solution. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

Correspondingly, the disclosure also provides an image processing device, which comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the image processing method according to any of the above embodiments, for example the method may comprise: acquiring an image to be processed and depth information thereof; performing background separation on the image to be processed based on the depth information to obtain a foreground image containing the shot subject and a background image not containing the shot subject; and carrying out fusion processing on the foreground image and the background image so that part of the shot main body in the fused image is not overlapped with the background image.

Accordingly, the present disclosure also provides an electronic device including a memory, and one or more programs, where the one or more programs are stored in the memory, and configured to be executed by the one or more processors, the one or more programs including instructions for implementing the image processing method according to any of the foregoing embodiments, for example, the method may include: acquiring an image to be processed and depth information thereof; performing background separation on the image to be processed based on the depth information to obtain a foreground image containing the shot subject and a background image not containing the shot subject; and carrying out fusion processing on the foreground image and the background image so that part of the shot main body in the fused image is not overlapped with the background image.

Fig. 12 is a block diagram illustrating an apparatus 1200 for implementing an image processing method according to an exemplary embodiment. For example, apparatus 1200 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like.

Referring to fig. 12, apparatus 1200 may include one or more of the following components: a processing component 1202, a memory 1204, a power component 1206, a multimedia component 1208, an audio component 1210, an input/output (I/O) interface 1212, a sensor component 1214, and a communications component 1216.

The processing component 1202 generally controls overall operation of the apparatus 1200, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 1202 may include one or more modules that facilitate interactions between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate interaction between the multimedia component 1208 and the processing component 1202.

The memory 1204 is configured to store various types of data to support operations at the apparatus 1200. Examples of such data include instructions for any application or method operating on the apparatus 1200, contact data, phonebook data, messages, pictures, videos, and the like. The memory 1204 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

Power supply assembly 1206 provides power to the various components of device 1200. The power supply components 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the apparatus 1200.

The multimedia component 1208 includes a screen between the device 1200 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1208 includes a front camera and/or a rear camera. The front camera and/or the rear camera may receive external multimedia data when the apparatus 1200 is in an operational mode, such as a photographing mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 1210 is configured to output and/or input audio signals. For example, the audio component 1210 includes a Microphone (MIC) configured to receive external audio signals when the apparatus 1200 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 1204 or transmitted via the communications component 1216. In some embodiments, audio assembly 1210 further includes a speaker for outputting audio signals.

The I/O interface 1212 provides an interface between the processing component 1202 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.

The sensor assembly 1214 includes one or more sensors for providing status assessment of various aspects of the apparatus 1200. For example, the sensor assembly 1214 may detect the on/off state of the device 1200, the relative positioning of the components, such as the display and keypad of the device 1200, the sensor assembly 1214 may also detect a change in position of the device 1200 or a component of the device 1200, the presence or absence of user contact with the device 1200, the orientation or acceleration/deceleration of the device 1200, and a change in temperature of the device 1200. The sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1214 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communications component 1216 is configured to facilitate communication between the apparatus 1200 and other devices, either wired or wireless. The apparatus 1200 may access a wireless network based on a communication standard, such as WiFi,2G or 3G,4G LTE, 5G NR (New Radio), or a combination thereof. In one exemplary embodiment, the communication component 1216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communications component 1216 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 1200 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.

In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, such as memory 1204, comprising instructions executable by processor 1220 of apparatus 1200 to perform the above-described method. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

The foregoing description of the preferred embodiments of the present disclosure is not intended to limit the disclosure, but rather to cover all modifications, equivalents, improvements and alternatives falling within the spirit and principles of the present disclosure.

Claims

1. An image processing method, comprising:

acquiring an image to be processed and depth information thereof;

2. The method according to claim 1, wherein the acquiring the image to be processed and the depth information thereof comprises:

3. The method of claim 1, wherein the background separating the image to be processed based on the depth information comprises:

4. A method according to claim 3, wherein determining the depth range of the position of the subject based on the proportion of each depth information in the image to be processed comprises:

5. The method as recited in claim 1, further comprising:

determining a middle-view area contained in the background image based on the depth information;

and blurring the pictures belonging to the middle-view area.

6. The method of claim 1, wherein the fusing the foreground image and the background image comprises:

7. The method of claim 1, wherein the fusing the foreground image and the background image comprises:

8. The method of claim 7, wherein said adjusting at least one of said foreground image and said background image comprises:

clipping the background image; and/or the number of the groups of groups,

and amplifying the foreground image.

9. The method of claim 8, wherein the cropping the background image comprises:

And clipping the background image based on the clipping template.

10. An image processing apparatus, comprising:

11. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to implement the method of any of claims 1-9 by executing the executable instructions.

12. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method according to any of claims 1-9.