CN110751668B - Image processing method, device, terminal, electronic equipment and readable storage medium - Google Patents

Abstract

The invention discloses an image processing method, device, terminal, electronic equipment, and computer-readable storage medium. The method includes: generating a disparity map according to a first image and a second image obtained by collecting images of the same target object ; When a preset operation performed by the user on the first image is detected, determining a target operation area targeted by the preset operation on the first image; based on the target operation area, the disparity map, and the The color value of each pixel point on the first image is used to extract the image corresponding to the target operation area from the first image. Since the main body area corresponding to the target operation area on the first image can be determined based on the disparity map, and then the image of the main body area is extracted, and the number of times the user marks the main body area to be extracted is small, the extracted The image of the main area improves the user experience.

Description

Image processing method, device, terminal, electronic device and readable storage medium

technical field

The present invention relates to the technical field of data processing, in particular to an image processing method, device, terminal, electronic equipment and a readable storage medium.

Background technique

The background replacement application supports users to extract the foreground subject area according to their own interests and replace the background arbitrarily. It is popular among users of Aikoutu (an image processing method used to extract the part of the image that the user wants from the original image). favorite. However, in related background replacement technologies, the extraction of the subject area usually requires the user to mark the subject area to be extracted multiple times or even mark the edge of the subject area. This operation is relatively cumbersome, and the user's ease of operation is not high. When it is necessary to extract the foreground of some colorful scenes, the extracted subject is often quite different from the subject specified by the user, resulting in poor user experience.

Contents of the invention

In view of the above problems, embodiments of the present invention are proposed to provide an image processing method, device, terminal, electronic device, and readable storage medium that overcome the above problems or at least partially solve the above problems.

The first aspect of the embodiments of the present invention provides an image processing method, the method comprising:

generating a disparity map according to the first image and the second image obtained by image acquisition of the same target object;

When detecting a preset operation performed by the user on the first image, determining a target operation area targeted by the preset operation on the first image;

An image corresponding to the target operation area is extracted from the first image based on the target operation area, the disparity map, and the color value of each pixel on the first image.

Optionally, extracting an image corresponding to the target operation area from the first image based on the target operation area, the disparity map, and the color values of each pixel on the first image includes:

Based on the target operation area and the disparity map, determine a preset disparity range corresponding to the target operation area on the disparity map;

Based on the preset disparity range, the position of the target operation area on the first image, and the color value of each pixel of the target operation area on the first image, perform a process on the disparity map segment, obtain and other regions other than said subject region;

Marking the main body area and other areas except the main body area to obtain a label map;

An image marked as a subject area on the label map is extracted from the first image.

Optionally, other areas except the subject area include: a non-subject area, a possible non-subject area, and a possible subject area; mark the subject area and other areas except the subject area to obtain a label Figures, including:

The non-subject area, the possible non-subject area, the possible subject area and the subject area are marked to obtain a label map.

Optionally, based on the preset parallax range, the position of the target operation area on the first image, and the color value of each pixel of the target operation area on the first image, the The disparity map is segmented, including:

determining each pixel in the disparity map that belongs to the preset disparity range;

Determining a plurality of first pixels whose color value difference between each pixel point and each pixel point of the target operation area is within a preset color value range, and is separated from the position of the target operation area by a preset distance points, determining an area formed by the plurality of first pixel points as the main body area;

In each of the pixels except the plurality of first pixels, it is determined that the difference between the color values of each pixel in the target operation area is within the preset color value range or, A plurality of second pixel points separated by a preset distance from the position of the target operation area, and an area formed by the plurality of second pixel points is determined as the possible subject area;

Determining the area formed by the remaining pixels in the pixels except the plurality of first pixels and the plurality of second pixels as the possible non-subject area;

An area formed by pixels in the disparity map that do not belong to the preset disparity range is determined as the non-subject area.

Optionally, after marking the non-subject area, the possible non-subject area, the possible subject area and the subject area to obtain a label map, the method further includes:

performing edge inpainting according to the label map and the first image, to determine an inpainted area belonging to the subject area;

re-marking the non-subject area, the possible non-subject area, the possible subject area, and the subject area based on the determined repair area belonging to the subject area;

Extracting the image marked as the main body area on the label map from the first image, including:

An image relabeled as a subject region on the label map is extracted from the first image.

Optionally, extracting an image marked as a subject area on the label map from the first image includes:

Determine the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area marked on the label map; wherein the alpha value of the non-subject area represents the The non-subject area is a transparent area, the alpha value of the subject area indicates that the subject area is an opaque area, the possible non-subject area and the alpha value of the possible subject area represent the possible non-subject area and the possible subject area is a translucent area;

Obtain an alpha image based on the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area on the label map;

An image of the subject area in the first image based on the color value of each pixel on the first image and the alpha image.

According to a second aspect of the embodiments of the present invention, an image processing device is provided, and the device includes:

A disparity map generating module, configured to generate a disparity map based on the first image and the second image obtained by image acquisition of the same target object;

A target operation area determination module, configured to determine a target operation area targeted by the preset operation on the first image when a preset operation performed by the user on the first image is detected;

An image extraction module, configured to extract an image corresponding to the target operation area from the first image based on the target operation area, the disparity map, and the color value of each pixel on the first image.

Optionally, the image extraction module includes:

A disparity range determination unit, configured to determine a preset disparity range corresponding to the target operation area on the disparity map based on the target operation area and the disparity map;

an area segmentation unit, configured to, based on the preset parallax range, the position of the target operation area on the first image, and the color value of each pixel of the target operation area on the first image, segmenting the disparity map to obtain a subject area and other areas except the subject area;

an area marking unit, configured to mark the subject area and other areas except the subject area to obtain a label map;

A main body area extraction unit, configured to extract an image marked as a main body area on the label map from the first image.

Optionally, other areas except the main body area include: a non-main body area, a possible non-main body area, and a possible main body area; The non-subject area, the possible subject area and the subject area are marked to obtain a label map.

The area segmentation unit includes:

A pixel determination subunit, configured to determine each pixel in the disparity map belonging to the preset disparity range;

The main body area determination subunit is configured to determine that the difference between the color values of each pixel point and each pixel point of the target operation area is within a preset color value range, and is separated from the position of the target operation area by a predetermined distance. Set a plurality of first pixel points at a distance, and determine an area formed by the plurality of first pixel points as the main body area;

The first unknown area determination subunit is configured to determine the difference between the color values of each pixel point in the target operation area and the color value of each pixel point in the target operation area in each pixel point except the plurality of first pixel points. A plurality of second pixel points within the preset color value range or a preset distance away from the position of the target operation area, determining an area formed by the plurality of second pixel points as the possible subject area ;

The second unknown area determination subunit is configured to determine, as the possible area formed by the remaining pixels in the respective pixels except the plurality of first pixels and the plurality of second pixels, non-subject area;

The non-subject area determining subunit is configured to determine, as the non-subject area, an area formed by pixels in the disparity map that do not belong to the preset disparity range.

Optionally, the device also includes:

An inpainting module, configured to perform edge inpainting according to the label map and the first image, so as to determine an inpainted area belonging to the subject area;

An update module, configured to remark the non-subject area, the possible non-subject area, the possible subject area, and the subject area based on the determined repair area belonging to the subject area;

The subject region extracting unit is specifically configured to extract an image relabeled as a subject region on the label map from the first image.

Optionally, the body region extraction unit includes:

an alpha value determining subunit, configured to determine the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area marked on the label map; wherein, the The alpha value of the non-subject area indicates that the non-subject area is a transparent area, the alpha value of the subject area indicates that the subject area is an opaque area, and the alpha values of the possible non-subject area and the possible subject area represent The possible non-subject area and the possible subject area are translucent areas;

an alpha image generating subunit, configured to obtain an alpha image based on the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area on the label map;

An image output subunit, configured to extract the image of the subject area in the first image based on the color value of each pixel on the first image and the alpha image.

The third aspect of the embodiments of the present invention provides an image processing terminal, including a display and an image processing device, the display is used to display a first image obtained by collecting images of the same target object, and the image processing device It is used to execute the image processing method described above.

According to a fourth aspect of the embodiments of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and operable on the processor. The processor implements the image processing method when executed.

According to a fifth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, and a computer program stored in the storage medium causes a processor to execute the image processing method.

Embodiments of the present invention include the following advantages:

In the embodiment of the present invention, the disparity map is generated based on the first image and the second image taken for the same target object, based on the disparity map and the target operation area clicked by the user on the first image, and each The color value of the pixel point extracts the image corresponding to the target operation area on the first image, and the image corresponding to the target operation area is the target image to be extracted by the user. Since the target image corresponding to the target operation area on the first image can be obtained based on the disparity map, the target image corresponding to the target operation area can be extracted, thereby reducing the number of times the user marks the subject area to be extracted, and then extracting The target image also improves the matching degree between the extracted image and the target image to which the target operation area designated by the user belongs, thereby improving user experience.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments of the present application. Obviously, the accompanying drawings in the following description are only some embodiments of the present application , for those skilled in the art, other drawings can also be obtained according to these drawings without paying creative labor.

Fig. 1 is a flow chart of the steps of an image processing method according to an embodiment of the present invention;

Fig. 2-A is the first image of an example of an image processing method in an embodiment of the invention;

Fig. 2-B is the tirmap figure of a kind of image processing method example of the embodiment of the invention;

Fig. 2-C is an alpha graph of an example of an image processing method of an embodiment of the invention;

FIG. 3 is a flow chart of steps for extracting an image corresponding to the target operation area in an image processing method according to an embodiment of the present invention;

FIG. 4 is a flow chart of steps for determining areas such as subject areas and non-subject areas in an image processing method according to an embodiment of the present invention;

Fig. 5 is a flow chart of the steps of extracting the image marked as the subject area on the label map from the first image in an image processing method according to an embodiment of the present invention;

FIG. 6 is an overall flowchart of an image processing method in an optional example of the present invention;

FIG. 7 is a schematic structural diagram of an image processing device according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

In related technologies, when extracting an image of a user-designated subject area from a target image, the user needs to click and confirm multiple times to extract the corresponding image, and the extracted image does not have a high matching degree with the user-designated subject area. The present disclosure provides the following image processing method, image processing device, electronic equipment, and computer-readable storage medium to solve the above-mentioned technical problems in related technologies. Below, the image processing method and image processing device are described respectively , an electronic device, and a computer-readable storage medium are described respectively.

Referring to FIG. 1 , it shows a flow chart of the steps of an image processing method according to an embodiment of the present invention. As shown in FIG. 1 , it may specifically include the following steps:

Step S11 , generating a disparity map according to the first image and the second image obtained by collecting images of the same target object.

In this embodiment, the same target object refers to the target to be photographed by the user, which may be a building, a landscape, an object or a person. A dual-camera module may be used to collect images of the target object, and the dual-camera module may be an intelligent collection device equipped with two cameras, for example, a mobile phone. The two cameras may be called a main camera and a sub camera respectively, where the lenses used by the main camera and the sub camera are different from each other. In practice, the main camera may use a telephoto lens, and the sub camera may use a wide angle lens. Wherein, the image of the target object captured by the main camera is called a first image, and the image of the target object captured by the secondary camera is called a second image.

Wherein, the disparity map generated based on the first image and the second image can be understood as an image based on the first image, whose size is the size of the first image, and whose element values are disparity values. The disparity value of each pixel in the disparity map can reflect the distance and direction information between the matching pixels on the first and second images, the disparity value of the same subject is similar, and the disparity value of different subjects may be There is a big difference. For example, taking a potted plant placed on a table as an example, the disparity values of the pixels included in the plant in the corresponding disparity map are not much different from each other. Compared with the pixels included in the table, the two The difference in disparity value is larger.

In an optional example, the disparity map can be generated using a 3D reconstruction algorithm in the related art. Optionally, an SGBM algorithm (semi-global block matching, semi-global block matching algorithm) can be used to generate the disparity map for the first image and the second The disparity map of the image, this algorithm is a lightweight 3D reconstruction algorithm, which has the characteristics of good parallax effect and fast speed, and can improve the fluency of generating the disparity map.

In step S12, when a preset operation performed by the user on the first image is detected, a target operation area targeted by the preset operation on the first image is determined.

In this embodiment, the first image can be stored and displayed, and the second image can be stored. In this way, the displayed first image can be viewed and operated by the user. The preset operation can be a click operation performed by the user with a mouse , or a remote control operation performed by holding a remote controller, or a finger touch operation. Specifically, when the user needs to extract an image of a certain area in the first image (hereinafter referred to as the target image) from the first image, he can use any of the above-mentioned preset operations to click any area, and then the clicked area can be determined as the target operation area. Wherein, the number of times to perform the preset operation may be one or more times.

For example, referring to FIG. 2-A , it shows an example diagram of the first image taken of the target plant in the natural environment. If the user wants to extract the plant image from the first image, then The mouse can be used to click on any area of the plant image on the first image, or perform a sliding touch once on any area of the plant image, such as leaves or branches. Wherein, the number of clicks may be one time.

Step S13 , based on the target operation area, the disparity map, and the color values of each pixel on the first image, extract an image corresponding to the target operation area from the first image.

The image corresponding to the target operation area is the image that the user needs to extract (that is, the target image). In the embodiment of the present invention, based on the clicked area, the disparity map, and the color value of each pixel on the first image, Extract the target image from the first image. Because the target image corresponding to the target operation area can be extracted based on the disparity map, the user can actually select any area in the target image to click once, and then the clicked area can be determined, because a clicked area is in the first image. may be part of a complete subject, and then determine the complete subject to which the clicked area belongs according to the characteristics of the disparity map, and then extract the image of the complete subject.

Exemplarily, taking Fig. 2-A as an example, assuming that the user clicks on the tree trunk on the plant image once, it is determined that the target operation area is the tree trunk or the leaves, and the tree trunk and the leaves belong to a part of the plant image in the first image, and then, can According to the characteristics of the disparity map, the corresponding plant image of the tree trunk in Fig. 2-A can be determined, and then the plant image can be transformed from Fig. 2-A according to the color value of each pixel included in the plant image in Fig. 2-A Extract it, that is, restore the color value of the plant image from Figure 2-A, and then realize the extraction of the plant image.

In the embodiment of the present invention, since the image corresponding to the target operation area is extracted based on the disparity map, images of the same subject with similar disparity and similar color can be extracted according to the disparity map, avoiding objects with similar colors but belonging to different subjects (such as plants and Table) are extracted together as a target, thereby improving the matching degree between the extracted image and the target image specified by the user, thereby improving user experience. Since the main image belonging to the target operation area can be determined through the target operation area clicked by the user, the number of target operation areas clicked can be reduced, and the number of times the user needs to mark the target image to be extracted can be further reduced, thereby further optimizing the user experience. .

Specifically, in one embodiment, as shown in FIG. 3 , it shows a flow chart of the steps of extracting the image corresponding to the target operation area in step S13, which may specifically include the following steps:

Step S131 , based on the target operation area and the disparity map, determine a preset disparity range corresponding to the target operation area on the disparity map.

In the embodiment of the present invention, the disparity map is generated based on the first image and the second image. In practice, the disparity map can run in the background, and the first image can be displayed on the front end for user operation. When the first image is determined After the target operation area is obtained, the position of the target operation area on the first image may be converted into a position on the disparity map, and then the disparity value of the target operation area on the disparity map is determined. Specifically, the position coordinates of the target operation region on the first image can be determined, and then according to the size ratio relationship between the first image and the disparity map, the position coordinates of the target operation region on the first image can be transformed into The position coordinates in , and then determine the position of the target operation area in the disparity map.

In practice, each pixel in the disparity map has its own disparity value, which can reflect the distance and direction information of the pixel. After the disparity value of the target operating area on the disparity map is determined, the preset disparity range targeted by the disparity value can be determined, specifically, the respective disparity values of each pixel in the disparity map within the preset disparity range The difference between the difference and the disparity value of the target operation region on the disparity map is smaller than a preset value. Wherein, the preset parallax range may be referred to as a reliable parallax range, and each pixel within the reliable parallax range has an overall correlation with each other in distance and direction from the pixel points of the target operation area, that is, within the reliable parallax range Each pixel of the target operation area may represent the same target image in practice.

Step S132, based on the preset parallax range, the position of the target operation area on the first image, and the color value of each pixel of the target operation area on the first image, the The disparity map is segmented to obtain the subject area and other areas except the subject area.

In the embodiment of the present invention, after the preset parallax range is determined, a plurality of pixel points within the preset parallax range and whose positions are close to the target operation area and have similar colors can be determined in the disparity map, and then the parallax Each pixel point determined on the figure is marked as the main body area, and the pixel points other than the determined multiple pixel points are marked as other areas.

For example, if the user clicks on the tree trunk, the preset disparity range of the tree trunk in the disparity map can be determined, and each pixel within the preset disparity range may belong to the same subject as the tree trunk, for example, the pixel points of the leaves Within the preset parallax range, the leaves and the trunk belong to the same plant, and the branches and leaves can be determined according to the position of the trunk on the first image and the color values of each pixel of the trunk in Fig. 2-A. , tree roots, etc., determine branches, leaves, roots, trunks, etc. as the main area, and other areas are determined as other areas.

Step S133, labeling the main body area and other areas except the main body area to obtain a label map.

In practice, after the main body area and other areas except the main body area are determined, an image segmentation algorithm based on graph theory can be used to obtain the above label map.

Step S134, extracting images marked as subject regions on the label map from the first image.

In this embodiment, each pixel point has been marked in the obtained label map, marking the main body area and other areas except the main body area, wherein the pixels in the main body area and the pixels of the target operation area If the points belong to the same target image, the image of the subject area can be extracted from the first image, so as to achieve the purpose of the user clicking the target operation area to extract the overall target image to which the target operation area belongs.

For example, take Figure 2-A as an example, the tree trunk clicked by the user, after the above steps, the main area in the label map is the plant image to which the tree trunk belongs, and then the entire plant image in Figure 2-A can be Extract it.

In an optional implementation manner, other areas except the main body area may include: a non-main body area, a possible non-main body area, and a possible main body area; correspondingly, in an optional implementation manner, The above step S133 may specifically be the following step S133':

Step S133', mark the non-subject area, the possible non-subject area, the possible subject area and the subject area to obtain a label map.

After the non-subject area, possible non-subject area, possible subject area and subject area are determined, the above label map can be obtained by using an image segmentation algorithm based on graph theory.

Optionally, the label map can be obtained by using grabcut in the related art. Specifically, when using grabcut, the first image and the non-subject area, the possible non-subject area, the possible subject area and the subject area are marked out. Taking an image as input, GrabCut can redistribute the labels of the above-mentioned regions according to the relationship between the labeled regions in the first image, and then obtain the label map. The image size of the label map obtained by using grabcut is relatively small, so it can improve the fluency of the image processing of this application on the low-end image processing terminal, and improve the processing speed.

Referring to Fig. 4, it is shown that other areas except the subject area may include: non-subject areas, possible non-subject areas, and possible subject areas, the steps of determining the above-mentioned subject area and the above-mentioned other areas The flow chart, as shown in Figure 4, may specifically include the following steps:

Step S21 , determining each pixel in the disparity map that belongs to the preset disparity range.

In practice, the difference between the disparity value of each pixel within the preset disparity range and the disparity value of the target operation area on the disparity map is smaller than a preset value. Wherein, the preset value may be set according to actual needs.

For example, taking Figure 2-A as an example, assuming that the user clicks on the tree trunk, each pixel within the preset disparity range corresponding to the tree trunk can be obtained in the disparity map, because each pixel in the disparity range is in the actual If it is possible to represent the same target image, the initial plant image to which the tree trunk belongs can be preliminarily determined, and then each pixel point of the initial plant image can be determined as each pixel point belonging to the preset parallax range.

Step S22, determining that the difference between the color values of each pixel point and each pixel point of the target operation area is within a preset color value range, and a plurality of color values are separated from the position of the target operation area by a preset distance. The first pixel, determining the area formed by the plurality of first pixels as the main body area.

In practice, the color value may refer to the value corresponding to the current color of the pixel point, and the lowest color value of each color is 0, and the highest value is 255. Similar colors have a small difference in color values, such as pink and red, and color values that differ in hue have a large difference, such as red and blue. In this way, the image formed by each pixel point within the preset color value range and each pixel point of the target operation area, both of which can represent the pixels of the same image on the first image, for example, both belong to the plant image .

Wherein, the position of the target operation region may refer to the position coordinates of the target operation region on the disparity map, specifically, the position coordinates of the target operation region on the disparity map may be determined according to the position of the target operation region on the first image, Furthermore, a plurality of pixel points separated by a preset distance from the position of the target operation region may be determined in the disparity map. In practice, the pixel points separated from the position of the target operation area by a preset distance represent the same target image as the target operation area in practice, and when reflected in the first image, the pixel points separated from the position of the target operation area by a preset distance The pixels of and the pixels of the target operation area all belong to the same target image, for example, both belong to the plant image.

In this embodiment, after each pixel point belonging to the preset parallax range is determined, in order to improve the accuracy of the extracted image, it is possible to further determine the target operation area among each pixel point belonging to the preset parallax range A plurality of first pixel points that are similar in color and position, for example, the pixel points of the branch area determined after clicking on the trunk. Furthermore, the target image corresponding to the target operation area can be identified, and the target image is the subject area.

Step S23, among the pixel points except the plurality of first pixel points, determine that the difference between the color values of each pixel point in the target operation area is within the preset color value range or, a plurality of second pixel points separated by a preset distance from the position of the target operation area, and an area formed by the plurality of second pixel points is determined as the possible subject area.

In practice, each pixel within the preset disparity range does not necessarily all belong to the same subject (that is, belongs to the target image in the first image). Therefore, after the first pixel is identified, the preset disparity can be Among the remaining pixels within the range, it is determined that the color value difference is within the preset color value range, or a plurality of second pixel points separated by a preset distance from the position of the target operation area, such as pixels in the leaf area point. Wherein, the color values of some of the plurality of second pixels and the target operation area are within a preset color value range, and the positions of another part of pixels are separated from the target operation area by a preset distance. The area formed by the plurality of second pixel points may be determined as a possible subject area, which means that the plurality of second pixel points may belong to the same target image as the target operation area.

Step S24 , determining an area formed by the remaining pixels in the respective pixels except the plurality of first pixels and the plurality of second pixels as the possible non-subject area.

Among them, the possible non-subject area indicates that the pixels in this area may not belong to the same target image as the target operation area.

It should be noted that, in practice, the possible non-subject area and the possible subject area may refer to the area between the subject area and the non-subject area, represented as the outline of the subject area. For example, taking FIG. 2-A as an example, the possible non-subject area and the possible subject area represent the enclosed area of the plant image.

Step S25, determining an area formed by pixels in the disparity map that do not belong to the preset disparity range as the non-subject area.

Wherein, the non-subject area indicates that the pixels in this area and the target operation area do not belong to the same target image.

After the non-subject area, possible non-subject area, possible subject area and subject area are determined by the above steps S21 to S25, the image corresponding to the target operation area can be extracted by following steps S133 to S134.

In one embodiment, referring to FIG. 5 , it shows a flowchart of the steps of extracting the image marked as the subject area on the label map from the first image in step S134, which may specifically include the following steps:

Step S31 , determining the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area and the subject area marked on the label map.

Wherein, the alpha value of the non-subject area indicates that the non-subject area is a transparent area, the alpha value of the subject area indicates that the subject area is an opaque area, and the possible non-subject area and the possible subject area The alpha value of indicates that the possible non-subject area and the possible subject area are translucent areas.

The alpha value refers to the value of the alpha channel, representing the transparency of the color. The value ranges from 0 to 1. The larger the value, the more opaque it is. For example, when the alpha value is 0, it is fully transparent; when the alpha value is 1, Indicates opacity, which is its true color. When the alpha value is a value between 0 and 1, it means translucent.

In the embodiment of this application, since the image of the subject area needs to be extracted, the original color of the image of the subject area needs to be fully reflected, while the images of other areas are not reflected. Therefore, the alpha value of the non-subject area can be determined If it is 0, it indicates that the non-subject area is fully transparent, and if the alpha value of the subject area is determined to be 1, it indicates that the subject area is completely opaque. The alpha value of the possible non-subject area and the possible subject area can be determined as a value between 0-1 by using the imagematting algorithm in the related art, which represents the semi-transparency of the possible area.

The specific principle of the image matting algorithm is calculated based on the possible subject area, the possible non-subject area and the subject area, the spatial distance and the color distance of the non-subject area. If the current pixel point is more similar to the non-subject area, the value of alpha closer to 0. For example, the more the alpha value of some pixels in the possible non-subject area tends to 0, the more transparent the pixel is; when the alpha value of some pixels in the possible subject area tends to 1, it indicates that the pixel is more transparent. Pixels tend to be more opaque.

Step S32: Obtain an alpha image based on the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area on the label map.

In the embodiment of the present invention, the alpha image may be referred to as an alpha channel map. Specifically, the alpha image may be obtained by using an existing related technology for generating an alpha channel map.

Step S33, based on the color value of each pixel on the first image and the alpha image, extract the image of the subject area in the first image.

In practice, the alpha channel and the three RGB channels together constitute a four-channel image. When extracting the image of the main body area, the alpha image can be input into the alpha channel, and the first image can be input into the three RGB channels. The alpha value of the main area is 1, and the alpha value of the non-subject area is 0. After the four-channel image is output, the color value of each pixel in the main area is the same as the color value displayed on the first image, that is, the main area The color value of each pixel is restored, and the non-subject area displays a masked image, that is, the color value of each pixel in the non-subject area is the background color, such as white or green. Since the alpha value of the possible subject area and the possible non-subject area is between 0 and 1, the color of each pixel in the two areas is displayed semi-transparently. In this way, the final output image shows the image of the subject area, because the pixels of possible subject areas and possible non-subject areas are displayed semi-transparently, so that the outline of the subject area is softened.

In practice, after the alpha channel and RGB three channels are output, the front-end display is the target image that the user needs to extract from the first image. Then, users can perform background replacement or edge refinement on the extracted image according to their own needs.

In one embodiment, after step S133 and before step S134, the following steps may also be included:

Step S133', perform edge inpainting according to the label map and the first image, so as to determine the inpainted area belonging to the subject area.

Step S133", based on the determined repair area belonging to the main body area, remark the non-body area, the possible non-body area, the possible main body area and the main body area.

In the embodiment of the present invention, the three-part graph trimap can be constructed based on the label map. Trimap is to classify the pixels in the image into three categories: the determined foreground, the determined background and the unknown area. The pixels in the unknown area are affected by the foreground pixels. Affected by background pixels. In this embodiment, the foreground area in the trimap obtained after the trimap is constructed is the subject area, the background area is the non-subject area, and the unknown area is the possible non-subject area and the possible subject area. After the trimap is constructed, It can be determined to perform edge repair on the possible non-subject area and the possible subject area on the label map, specifically, edge repair can be performed based on trimap and shared-samplingbasedmatting algorithm, so as to combine the possible non-subject area and the possible subject area The area belonging to the main body area in the area (that is, the repair area described in the embodiment of the present invention) is determined as the main body area.

When the above technical solution is adopted, since the trimap is constructed based on the label map, the edge contour of the main body area is finer, and the contour of the finally extracted image of the main body area is also finer, which further improves the extracted image and the original first. The degree of matching and realism of the target image on the image.

For example, taking Figure 2-A as an example, the trimap image constructed from the label image is shown in Figure 2-B, and the final alpha image obtained based on the trimap image, color image and edge repair is shown in Figure 2-C It can be seen that after edge restoration, the edge profile of the alpha image obtained in 2-C is very fine, so that the final output plant image matches the plant image in the original Figure 2-A more closely, and the degree of restoration is higher.

Correspondingly, the above step S134 can use the method described in the following steps to extract the image of the subject area:

Step S134', extracting from the first image the image relabeled as the subject area on the label map.

The specific content of the step S134' is similar to that of the step S134, and details can be referred to the step S134, which will not be repeated here.

Referring to FIG. 6 , it shows a complete flow chart of an optional example image processing method. This option is described by extracting a plant image from the first image shown in FIG. 2-A as an example, wherein, The second image taken for the target object in Fig. 2-A is cached in memory, and the specific process is as follows:

First, for the first image and the second image, a classic 3D reconstruction algorithm, such as the SGBM algorithm, is used to generate a disparity map.

Secondly, determine the target operation area clicked by the user on the first image, extract the subject area, non-subject area, possibly the subject area (that is, the possible subject area) and the possibly non-subject area (possible non-subject area) .

Then, the image segmentation algorithm based on graph theory, such as the Grabcut algorithm, is used for rough segmentation to obtain the label map.

Then, construct a trimap based on the label map, and obtain a trimap map as shown in Figure 2-B, where the subject area in the label map is the foreground area in the trimap map, and the non-subject area is the background area in the trimap map. The subject area and possible non-subject area are the unknown areas in the trimap map, that is, the grayscale area in the map.

Then, based on the trimap, the edge repair of the main body area is further carried out, and the repair area belonging to the main body area among the possible main body area and the possible non-subject area is marked as the main body area, and then the repair of the unknown area in the trimap map is realized, that is, is the edge repair, and based on the repaired result, an alpha image is finally obtained. In the alpha image, the alpha value of the subject area is 1, the alpha value of the non-subject area is 0, and the alpha value of the unknown area (including possible subject areas and possible non-subject areas) can be calculated as 0-1 by using image matting value between.

Finally, the alpha image is input to the alpha channel (that is, the alpha image is output as the fourth channel of the color map as described in Figure 6), and the first image is input to the RGB channel, thereby displaying the image of the main body area on the display screen. That is, the plant image in Fig. 2-A is displayed.

It should be noted that, for the method embodiment, for the sake of simple description, it is expressed as a series of action combinations, but those skilled in the art should know that the embodiment of the present invention is not limited by the described action sequence, because According to the embodiment of the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

Referring to FIG. 7, an image processing device according to an embodiment of the present invention is shown, and the device may specifically include the following modules:

A disparity map generating module 71, configured to generate a disparity map according to the first image and the second image obtained by image acquisition of the same target object;

A target operation area determination module 72, configured to determine the target operation area targeted by the preset operation on the first image when a preset operation performed by the user on the first image is detected;

An image extraction module 73, configured to extract an image corresponding to the target operation area from the first image based on the target operation area, the disparity map, and the color value of each pixel on the first image.

In one embodiment, the image extraction module may include:

A disparity range determination unit, configured to determine a preset disparity range corresponding to the target operation area on the disparity map based on the target operation area and the disparity map;

an area segmentation unit, configured to, based on the preset parallax range, the position of the target operation area on the first image, and the color value of each pixel of the target operation area on the first image, segmenting the disparity map to obtain a subject area and other areas except the subject area;

an area marking unit, configured to mark the subject area and other areas except the subject area to obtain a label map;

A main body area extraction unit, configured to extract an image marked as a main body area on the label map from the first image.

In one embodiment, other areas except the main body area may include: a non-main body area, a possible non-main body area, and a possible main body area; the area marking unit is specifically used to mark the non-main body area, The possible non-subject area, the possible subject area and the subject area are marked to obtain a label map.

The area segmentation unit may include:

A pixel determination subunit, configured to determine each pixel in the disparity map belonging to the preset disparity range;

The main body area determination subunit is configured to determine that the difference between the color values of each pixel point and each pixel point of the target operation area is within a preset color value range, and is separated from the position of the target operation area by a predetermined distance. Set a plurality of first pixel points at a distance, and determine an area formed by the plurality of first pixel points as the main body area;

The first unknown area determination subunit is configured to determine the difference between the color values of each pixel point in the target operation area and the color value of each pixel point in the target operation area in each pixel point except the plurality of first pixel points. A plurality of second pixel points within the preset color value range or a preset distance away from the position of the target operation area, determining an area formed by the plurality of second pixel points as the possible subject area ;

The second unknown area determination subunit is configured to determine, as the possible area formed by the remaining pixels in the respective pixels except the plurality of first pixels and the plurality of second pixels, non-subject area;

The non-subject area determining subunit is configured to determine, as the non-subject area, an area formed by pixels in the disparity map that do not belong to the preset disparity range.

In one embodiment, the device may also include the following modules:

An inpainting module, configured to perform edge inpainting according to the label map and the first image, so as to determine an inpainted area belonging to the subject area;

An update module, configured to remark the non-subject area, the possible non-subject area, the possible subject area, and the subject area based on the determined repair area belonging to the subject area;

The subject region extracting unit is specifically configured to extract an image relabeled as a subject region on the label map from the first image.

In one embodiment, the body region extraction unit may include:

an alpha value determining subunit, configured to determine the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area marked on the label map; wherein, the The alpha value of the non-subject area indicates that the non-subject area is a transparent area, the alpha value of the subject area indicates that the subject area is an opaque area, and the alpha values of the possible non-subject area and the possible subject area represent The possible non-subject area and the possible subject area are translucent areas;

an alpha image generating subunit, configured to obtain an alpha image based on the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area on the label map;

An image output subunit, configured to extract the image of the subject area in the first image based on the color value of each pixel on the first image and the alpha image.

As for the embodiment of the image processing apparatus, since it is basically similar to the embodiment of the image processing method, the description is relatively simple, and for related parts, refer to the part of the description of the embodiment of the image processing method.

An embodiment of the present invention also provides an image processing terminal, including a display and an image processing device, the display is used to display the first image obtained by collecting images of the same target object, and the image processing device is used to execute The image processing method described in the above-mentioned embodiments.

Referring to FIG. 8 , a schematic structural diagram of an electronic device 800 according to an embodiment of the present invention is shown. The electronic device 800 can be used for image processing, and can include a memory 81, a processor 82, and can be stored in the memory 81 and can be processed. The computer program running on the processor, the processor 82 is configured to execute the image processing method.

An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored to enable a processor to execute the image processing method.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, devices, or computer program products. Accordingly, embodiments of the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Embodiments of the present invention are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor or processor of other programmable data processing terminal equipment to produce a machine such that instructions executed by the computer or processor of other programmable data processing terminal equipment Produce means for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the The instruction means implements the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded into a computer or other programmable data processing terminal equipment, so that a series of operational steps are performed on the computer or other programmable terminal equipment to produce computer-implemented processing, thereby The instructions executed above provide steps for implementing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Having described preferred embodiments of embodiments of the present invention, additional changes and modifications to these embodiments can be made by those skilled in the art once the basic inventive concept is appreciated. Therefore, the appended claims are intended to be construed to cover the preferred embodiment and all changes and modifications which fall within the scope of the embodiments of the present invention.

Finally, it should also be noted that in this text, relational terms such as first and second etc. are only used to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations, any such actual relationship or order exists. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or terminal equipment comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements identified, or also include elements inherent in such a process, method, article, or end-equipment. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising said element.

The image processing method, device, terminal, electronic equipment and readable storage medium provided by the present invention have been introduced in detail above. In this paper, specific examples are used to illustrate the principle and implementation of the present invention. The above embodiments The description is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and scope of application. In summary , the contents of this specification should not be construed as limiting the present invention.

Claims (9)

1. An image processing method, characterized in that the method comprises: generating a disparity map according to the first image and the second image obtained by image acquisition of the same target object; When detecting a preset operation performed by the user on the first image, determining a target operation area targeted by the preset operation on the first image; Perform edge inpainting according to the label map and the first image to determine the inpainted area belonging to the main body area, and based on the determined inpainted area belonging to the main body area, redo the non-subject area, possible non-subject area, and possible subject area Marking the area and the main body area, wherein the performing edge restoration according to the label map and the first image is determining to perform edge repair on the possible non-subject area and the possible subject area on the label map; Extracting an image corresponding to the target operation area from the first image based on the target operation area, the disparity map, and the color values of each pixel on the first image, including: Based on the target operation area and the disparity map, determine a preset disparity range corresponding to the target operation area on the disparity map; Based on the preset disparity range, the position of the target operation area on the first image, and the color value of each pixel of the target operation area on the first image, perform a process on the disparity map Segmentation to obtain the main body area and other areas except the main body area; Marking the main body area and other areas except the main body area to obtain a label map; An image marked as a subject area on the label map is extracted from the first image. 2. The method according to claim 1, characterized in that, other regions except the subject region include: non-subject regions, possible non-subject regions, and possible subject regions; Other areas outside the main area are marked to obtain a label map, including: The non-subject area, the possible non-subject area, the possible subject area and the subject area are marked to obtain a label map. 3. The method according to claim 2, characterized in that based on the preset parallax range, the position of the target operation area on the first image, and the target operation on the first image The color value of each pixel in the region is used to segment the disparity map, including: determining each pixel in the disparity map that belongs to the preset disparity range; Determining a plurality of first pixels whose color value difference between each pixel point and each pixel point of the target operation area is within a preset color value range, and is separated from the position of the target operation area by a preset distance points, determining an area formed by the plurality of first pixel points as the main body area; In each of the pixels except the plurality of first pixels, it is determined that the difference between the color values of each pixel in the target operation area is within the preset color value range or, A plurality of second pixel points separated by a preset distance from the position of the target operation area, and an area formed by the plurality of second pixel points is determined as the possible subject area; Determining the area formed by the remaining pixels in the pixels except the plurality of first pixels and the plurality of second pixels as the possible non-subject area; An area formed by pixels in the disparity map that do not belong to the preset disparity range is determined as the non-subject area. 4. The method according to claim 2 or 3, characterized in that, after marking the non-subject area, the possible non-subject area, the possible subject area and the subject area, a label map is obtained Afterwards, the method also includes: performing edge inpainting according to the label map and the first image, to determine an inpainted area belonging to the subject area; re-marking the non-subject area, the possible non-subject area, the possible subject area, and the subject area based on the determined repair area belonging to the subject area; Extracting the image marked as the main body area on the label map from the first image, including: An image relabeled as a subject region on the label map is extracted from the first image. 5. The method according to claim 2 or 3, wherein extracting the image marked as the subject area on the label map from the first image comprises: Determine the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area marked on the label map; wherein the alpha value of the non-subject area represents the The non-subject area is a transparent area, the alpha value of the subject area indicates that the subject area is an opaque area, the possible non-subject area and the alpha value of the possible subject area represent the possible non-subject area and The possible subject area is a translucent area; Obtain an alpha image based on the respective alpha values of the non-subject area, the possible non-subject area, the possible subject area, and the subject area on the label map; Based on the color value of each pixel on the first image and the alpha image, extract the image of the subject area in the first image. 6. An image processing device, characterized in that the device comprises: A disparity map generating module, configured to generate a disparity map based on the first image and the second image obtained by image acquisition of the same target object; A target operation area determination module, configured to determine a target operation area targeted by the preset operation on the first image when a preset operation performed by the user on the first image is detected; The inpainting module is configured to perform edge inpainting according to the label map and the first image, so as to determine the inpainted area belonging to the subject area, and based on the determined inpainted area belonging to the subject area, to redo the non-subject area and the possible non-subject area regions, possible subject regions, and the subject region, wherein the edge restoration according to the label map and the first image is to determine the possible non-subject regions and the possible subject areas on the label map edge repair in the main body area; An image extraction module, configured to extract an image corresponding to the target operation area from the first image based on the target operation area, the disparity map, and the color value of each pixel on the first image, the Image extraction modules include: A disparity range determination unit, configured to determine a preset disparity range corresponding to the target operation area on the disparity map based on the target operation area and the disparity map; an area segmentation unit, configured to, based on the preset parallax range, the position of the target operation area on the first image, and the color value of each pixel of the target operation area on the first image, segmenting the disparity map to obtain a subject area and other areas except the subject area; an area marking unit, configured to mark the subject area and other areas except the subject area to obtain a label map; A main body area extraction unit, configured to extract an image marked as a main body area on the label map from the first image. 7. An image processing terminal, characterized in that it comprises a display and an image processing device, the display is used to display the first image obtained by image acquisition of the same target object, and the image processing device is used to perform the following steps: The image processing method described in any one of claims 1-5. 8. An electronic device, characterized in that it comprises a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements any of claims 1-5 when executed. image processing method. 9. A computer-readable storage medium, wherein a computer program stored therein enables a processor to execute the image processing method according to any one of claims 1-5.

Images