CN106469446B

CN106469446B - Depth image segmentation method and segmentation device

Info

Publication number: CN106469446B
Application number: CN201510520359.6A
Authority: CN
Inventors: 吴小勇; 刘洁; 王维
Original assignee: Xiaomi Inc
Current assignee: Xiaomi Inc
Priority date: 2015-08-21
Filing date: 2015-08-21
Publication date: 2021-04-20
Anticipated expiration: 2035-08-21
Also published as: CN106469446A

Abstract

The present disclosure relates to a depth image segmentation method and a depth image segmentation apparatus. The method comprises the following steps: obtaining the depth value of a pixel point in a first selected area in the depth image; estimating a depth value range of the segmentation object in the first selected area; determining pixel points of the first selected area, the depth values of which are within the depth value range, and forming a depth value export area according to the determined pixel points; and segmenting the segmentation object in the first selection area according to the depth value derivation area to obtain the segmentation object in the first selection area. The depth image segmentation method and the segmentation device provided by the disclosure can accurately segment a required object when the depth difference between a segmented object and a background is large.

Description

Depth image segmentation method and segmentation device

Technical Field

The present disclosure relates to the field of computer vision, and in particular, to a depth image segmentation method and a depth image segmentation apparatus.

Background

Image segmentation refers to the process of dividing a digital image into a plurality of image regions (sets of pixel points). Conventional image segmentation methods are based on processing pixel values in an image. Therefore, the segmentation effect is more desirable in a place where the color difference between the segmentation object and the background is large, and the segmentation effect is less desirable in a place where the color difference between the segmentation object and the background is small.

Disclosure of Invention

To overcome the problems in the related art, the present disclosure provides a depth image segmentation method and a depth image segmentation apparatus.

The inventors have conceived that there is currently a type of image data including depth information of an image. When the depth difference between the segmentation object and the background is large, the image segmentation is carried out by using the depth information, so that the problem of unclear boundaries caused by segmentation according to pixel values in the related art can be solved, and the segmentation result is more accurate.

According to a first aspect of embodiments of the present disclosure, a method for segmenting a depth image is provided. The method comprises the following steps: obtaining the depth value of a pixel point in a first selected area in the depth image; estimating a depth value range of the segmentation object in the first selected area; determining pixel points of the first selected area, the depth values of which are within the depth value range, and forming a depth value export area according to the determined pixel points; and segmenting the segmentation object in the first selection area according to the depth value derivation area to obtain the segmentation object in the first selection area.

According to a second aspect of the embodiments of the present disclosure, there is provided a depth image segmentation apparatus. The device comprises: the depth value acquisition module is configured to acquire the depth value of a pixel point in a first selected area in the depth image; a depth value range estimation module configured to estimate a depth value range of the segmented object in the first selected region; a depth value derivation area determination module configured to determine pixel points in the first selected area, the depth values of which are within the depth value range, and form a depth value derivation area according to the determined pixel points; and a first segmentation module configured to segment the segmented object in the first selected area according to the depth value derivation area, so as to obtain the segmented object in the first selected area.

According to a third aspect of the embodiments of the present disclosure, there is provided a depth image segmentation apparatus. The device comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: obtaining the depth value of a pixel point in a first selected area in the depth image; estimating a depth value range of the segmentation object in the first selected area; determining pixel points of the first selected area, the depth values of which are within the depth value range, and forming a depth value export area according to the determined pixel points; and segmenting the segmentation object in the first selection area according to the depth value derivation area to obtain the segmentation object in the first selection area.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:

the required object is segmented in the image according to the depth information acquired in the shooting process, large effective information blocks cannot be lost in the segmented image, large redundant information blocks cannot occur, and the edge of image segmentation is finer. Therefore, the depth image segmentation method and the segmentation device provided by the disclosure can accurately segment the required segmentation object when the depth difference between the segmentation object and the background is large.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a schematic diagram of an image to be segmented, according to an exemplary embodiment;

FIG. 2 is a flow diagram illustrating a method of segmentation of a depth image in accordance with an exemplary embodiment;

FIG. 3 is a schematic illustration of a first selected area of FIG. 1 shown in accordance with an exemplary embodiment;

FIG. 4 is a flowchart illustrating estimating a range of depth values for a segmented object in a first selected region in accordance with an illustrative embodiment;

FIG. 5 is a histogram of the distribution of depth values for all of the pixels in FIG. 1;

FIG. 6 is a flowchart illustrating estimating a range of depth values for a segmented object in a first selected region in accordance with another illustrative embodiment;

FIG. 7 is a schematic illustration of a reference region shown in accordance with an exemplary embodiment;

FIG. 8 is an interface diagram illustrating a dialog box for estimating a range of depth values for a segmented object in a first selection region in accordance with an illustrative embodiment;

FIG. 9 is a flowchart illustrating a method of segmentation of a depth image in accordance with another exemplary embodiment;

FIG. 10 is a flowchart illustrating a method of segmentation of a depth image in accordance with yet another exemplary embodiment;

FIG. 11 is a diagram illustrating the formation of segmented objects in a depth image according to an exemplary embodiment;

FIG. 12 is a schematic diagram illustrating the formation of segmented objects in a depth image according to another exemplary embodiment;

FIG. 13 is a block diagram illustrating a depth image segmentation apparatus in accordance with an exemplary embodiment;

FIG. 14 is a block diagram illustrating the structure of a depth value range estimation module in accordance with an exemplary embodiment;

FIG. 15 is a block diagram of a depth value range estimation module shown in accordance with another exemplary embodiment;

fig. 16 is a block diagram illustrating a depth image segmentation apparatus according to another exemplary embodiment;

fig. 17 is a block diagram illustrating a depth image segmentation apparatus according to another exemplary embodiment; and

fig. 18 is a block diagram illustrating a depth image segmentation apparatus according to an exemplary embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

FIG. 1 is a schematic diagram illustrating an image to be segmented according to an exemplary embodiment. In the image shown in fig. 1, a cat walking on the roof was photographed, and the background was a blue sky and a white cloud. All we need to do is to segment this cat from the whole image. That is, the segmentation object in fig. 1 is a cat. If the cat is a black cat or a yellow cat, the segmentation effect is better by using the related technology. If the cat is a white cat, the color of the segmentation object (white cat) is the same as that of the background (white cloud), and therefore, when segmentation is performed by using the correlation technique, the boundary is not clear, and the segmentation effect is not good.

Then how to distinguish when the color of the segmentation object is similar to that of the background can make the segmentation effect better. The inventors contemplate being able to make use of depth information in the image for discrimination.

In the related art of computer vision, a "motion sensing camera" (also called a depth camera) capable of capturing motion sensing has been developed, and the motion sensing camera can acquire distance information indicating each point in a captured picture and the camera when capturing the image. For example, the magnificent Xtion PRO camera. Microsoft corporation also applies the somatosensory camera to Xbox ONE game machines developed by Microsoft corporation. In an image (simply referred to as a depth image) photographed by such a motion sensing camera, each pixel point has depth information in addition to a pixel value. The depth information of a pixel point is information indicating the distance between the position of the pixel point in the picture and the camera when the image is shot.

That is, if the image in fig. 1 is a depth image captured by a depth camera, the white cat and the white cloud can be distinguished according to the difference of the depth information of the white cat and the white cloud, so as to achieve a better segmentation effect. Therefore, the inventors provide a depth image segmentation method and a depth image segmentation device according to the present disclosure based on the above inventive concepts. The depth image segmentation method and the depth image segmentation apparatus provided by the present disclosure are described in detail below.

FIG. 2 is a flow chart illustrating a method of segmentation of a depth image in accordance with an exemplary embodiment. As shown in fig. 2, the method includes the following steps.

In step S11, the depth value of the pixel point in the first selected region in the depth image is obtained.

First, a first selection area is selected from the depth image. The range of the first selected area may be determined as the case may be, and may be the entire depth image or a part of the depth image. In the case where the approximate range of the segmentation object in the depth image can be determined, a part of the depth image may be selected as the first selected region so that the segmentation object of the depth image is contained in the first selected region. And processing only the pixel points in the first selected region to obtain a segmentation object in the first selected region, namely obtaining the segmentation object of the whole depth image.

For example, FIG. 3 is a schematic illustration of a first selected area of FIG. 1 shown in accordance with an exemplary embodiment. As shown in fig. 3, to segment the image of the white cat, a dashed rectangle may be selected as the first selected area. The white cat to be split is enclosed in a dashed rectangle. In this way, when a part of the depth image is selected as the first selected region, the range of the image region to be processed can be narrowed down, the amount of computation can be reduced, and the speed of segmentation can be increased.

In a depth image shot by a depth camera, each pixel point has depth information. The distance (depth) between the position of each pixel point in the picture and the camera is theoretically zero to infinity. In practical application, the depth of each pixel point can be normalized to obtain the depth value in any range. That is, in the depth image captured by the depth camera, a numerical value (depth value) indicating depth information associated with each pixel point may be acquired.

In the step S12, a range of depth values of the segmented object in the first selected region is estimated.

To segment the segmented object in the first selected region from the first selected region, it is necessary to determine which pixels belong to the segmented object and which pixels do not belong to the segmented object. In step S13, a depth value range of the pixel points included in the segmentation object may be estimated, and then the pixel points within the depth value range may be found according to the estimated depth value range.

As described above, the depth value may be a numerical value within an arbitrary range after being subjected to the normalization process. The range of depth values of the segmentation object can be estimated according to the depth values of the pixel points in the first selected region acquired in step S11.

Specifically, the depth value distribution conditions of all the pixel points in the first selected region may be counted first, and the depth value range of the segmentation object in the first selected region is determined according to the distribution conditions. FIG. 4 is a flowchart illustrating estimating a range of depth values for a segmented object in a first selected region in accordance with an exemplary embodiment. As shown in fig. 4, estimating the range of depth values of the split object in the first selected area (step S12) includes the following steps.

In step S121, the distribution status of the depth values of all the pixels in the first selected region is determined according to the obtained depth values of the pixels in the first selected region.

The distribution condition of the depth values reflects the proportion relation of all the depth values in the first selected area. The distribution of the depth values may be represented by, for example, the number of pixels of each depth value in each depth value range, and the like, and may be counted in various forms, such as a graph, a histogram, and the like. The distribution conditions of the depth values of all the pixel points in the first selected area are mastered, and the depth value range of the segmentation object in the first selected area can be estimated by combining the specific scene in the depth image.

In step S122, a range of depth values of the segmentation object in the first selected area is estimated according to the determined distribution condition.

The depth value range of the segmented object in the first selected area may be estimated based on the estimated proportion relationship between the segmented object in the first selected area and the depth difference between the segmented object in the first selected area and the background.

For example, fig. 5 is a histogram of the distribution of depth values of all the pixels in fig. 1. As shown in fig. 5, the X-axis represents the depth values of the pixels, and the depth values are normalized and distributed between 0 and 200. The Y-axis represents the number or proportion of pixel points. The scenario of fig. 1 contains a cat and a background of the sky. It is estimated that the cat occupies about half the area of the sky, and the depth value of the sky should be the maximum value of the depth values. When the first selected area is selected as the whole depth image, it can be determined that the depth values in the first selected area are substantially concentrated in two depth value areas. The cat contains pixel points whose depth values are concentrated in one depth value region, and the sky pixel values are concentrated in another depth value region (near the maximum depth value). It can be determined from the histogram of fig. 5 that the depth value of the pixel corresponding to the cat in fig. 1 should be between 40-80, and most other areas (sky and white cloud) correspond to a depth value close to 200.

In step S12, a reference area may be first selected in the first selected area, and the range of depth values of the segmented object in the first selected area is estimated according to the range of depth values in the selected reference area. FIG. 6 is a flowchart illustrating estimating a range of depth values for a segmented object in a first selected region in accordance with another illustrative embodiment. As shown in fig. 6, estimating the range of depth values of the split object in the first selected area (step S12) includes the following steps.

In step S123, a reference area is selected from the first selected area.

Selecting the reference area is intended to estimate a range of depth values of the segmented object in the first selected area from a range of depth values of the reference area. Thus, the reference area may select an area having a close association in depth value with the segmented object. For example, a part of the segmented object in the first selected region may be used as the reference region.

FIG. 7 is a schematic diagram illustrating a reference region in accordance with an exemplary embodiment. As shown in fig. 7, a first selected area may be selected as the whole image, and a part of the area I on the cat body may be selected as the reference area I. The user determines the reference area I as part of the segmented object (cat). Then, the range of depth values of the whole cat is determined according to the range of depth values of the reference area I, so that the whole cat is segmented from the image, and the following steps are described in detail below.

In step S124, a depth value range of the reference area is determined.

According to the depth values of the pixel points in the first selected region in the depth image obtained in step S11, a depth value range in any selected region in the first selected region can be obtained.

In step S125, a depth value range of the segmentation object in the first selected area is estimated according to the depth value range of the reference area.

That is, the range of depth values of the division object in the first selected area is estimated from the range of depth values of the reference area in combination with the scene of the image (the relationship of the reference area and the division object). For example, the range of depth values of the entire cat (the split object in the first selected region) is judged from the depth ranges 50 to 55 of the reference region I of the split object cat in fig. 7.

In the case where the selected reference area belongs to a partial area of the segmented object in the first selected area, the depth value range of the segmented object in the first selected area may be determined according to the position, size, and the like of the reference area in the segmented object. For example, the reference area I in fig. 7 is selected as an area on the cat body, and the depth range thereof is 50-55, and since the reference area I on the cat body is located at the comparison front end of the whole cat with respect to the camera, the depth value range of the whole cat (segmentation object) can be determined therefrom to be, for example, 40-80.

In the software interface for depth image segmentation, an interactive interface may be provided for estimating a depth value range of a segmentation object in the first selected region from a depth value range of the reference region, for selection by a user. FIG. 8 is an interface diagram illustrating a dialog box for estimating a range of depth values for a segmented object in a first selection region in accordance with an illustrative embodiment. As shown in fig. 8, in step S125, when the depth value range of the division object in the first selected area is determined according to the depth value range of the reference area, this dialog box may pop up. Wherein the determined depth value range of the reference area may be marked with a box. The reference area is provided with slidable arrows at the front and the back, and the user can input the estimated depth value range of the segmentation object in the first selected area in a sliding manner. In the dialog box shown in fig. 8, the depth range of the reference region is 50-55 and the depth value range of the estimated segmentation object is 40-80.

When image segmentation is performed, it is often necessary to segment a human image. In the embodiment shown in fig. 6, in the case where the segmentation object in the first selected region is a human image, the reference region may be selected as a skin color region. That is, a pixel point of the skin color in the first selected region may be detected, and the detected skin color region including the pixel point of the skin color is used as a reference region.

For example, in a software interface for depth image segmentation, a "choose skin tone" button may be set in the toolbar. By clicking on the button, a skin tone (e.g., yellow skin) identification of the first selected area may be automatically performed by the computer.

And after the skin color area in the first selected area is identified, the skin color area is used as a reference area. The range of depth values of the person image may be estimated in the dialog box shown in fig. 8 based on the position of the skin color region in the person image, thereby determining the entire region of the person image in the first selected region.

The skin color area is selected as the reference area, so that the skin color area can be automatically detected by a computer, and the trouble of manually selecting the reference area is avoided. Moreover, the selected reference area has a typical representativeness in the segmented object, which makes the depth value range of the segmented object determined next more accurate, and thus makes the segmentation result of the human image more accurate.

After the depth value range of the segmentation object in the first selection area is estimated, the corresponding pixel point can be found according to the estimated depth value range, and therefore image segmentation is carried out.

In step S13, a pixel point in the first selected region having a depth value within the depth value range is determined, and a depth value derivation region is formed according to the determined pixel point.

Wherein, the depth value derivation area is an area derived according to a depth value mode. In the user's operation interface, the dialog box shown in fig. 8 may be displayed simultaneously with the depth image on the display. And displaying the determined depth value derivation area in the depth image while the user slides the arrow in the dialog box, so that the user can conveniently adjust the depth value range of the segmentation object in the first selected area according to the determined depth value derivation area, and the optimal segmentation result is obtained.

In step S14, the segmented object in the first selected area is segmented according to the depth value derivation area, so as to obtain the segmented object in the first selected area. That is, the depth value deriving area is segmented from the depth image to form a segmentation object in the first selected area.

According to the method and the device, the required object is segmented in the image according to the depth information acquired in the shooting process, large effective information blocks cannot be lost in the segmented image, large redundant information blocks cannot occur, and the edge of image segmentation is finer. Therefore, the depth image segmentation method provided by the disclosure can accurately segment the required segmented object when the depth difference between the segmented object and the background is large.

Optionally, the depth image segmentation method provided by the above embodiment of the present disclosure is combined with a conventional segmentation method, and respective advantages are selected, so that the obtained segmentation result is more accurate than the segmentation result obtained by using only one of the methods.

Fig. 9 is a flowchart illustrating a method of segmentation of a depth image according to another exemplary embodiment. As shown in fig. 9, the method further comprises the following steps based on the embodiment shown in fig. 2.

In step S15, pixel values of pixel points in the second selected region in the depth image are obtained.

The purpose of selecting the second selected area in the depth image is to apply a conventional segmentation method for segmentation in a later step. The second selected area may be the entire depth image or a portion of the depth image. The second selection area and the first selection area may or may not overlap each other. For example, when the entire depth image is to be segmented by the conventional method, the entire depth image may be selected as the second selected region, and in this case, the segmented region obtained by the conventional segmentation method may be combined with the depth value derivation regions obtained in steps S11 to S14, resulting in a more accurate segmentation result than that obtained by only one method. Several embodiments of fusing the two methods are described in detail below.

In step S16, the pixel points of the segmented object in the second selected region are determined according to the pixel values of the pixel points in the second selected region, and a pixel value derivation region is formed according to the determined pixel points.

The pixel value derivation area is an area derived in a pixel value manner. The conventional segmentation method is to process the pixel values to obtain a pixel value derivation area. In step S18, the segmentation is performed by a conventional method. Including any one of the following: determining pixel points of the segmentation objects in the second selected region according to a threshold-based segmentation method, and forming a pixel value derivation region according to the determined pixel points; determining pixel points of the segmentation object in the second selected region according to an edge-based segmentation method, and forming a pixel value derivation region according to the determined pixel points; determining pixel points of the segmentation object in the second selected region according to a region-based segmentation method, and forming a pixel value derivation region according to the determined pixel points; determining pixel points of the segmentation object in the second selected region according to a graph theory-based segmentation method, and forming a pixel value derivation region according to the determined pixel points; or determining pixel points of the segmentation object in the second selection region according to a segmentation method based on the energy functional, and forming a pixel value derivation region according to the determined pixel points. The above various methods are well known to those skilled in the art and will not be described in detail herein.

In step S17, the segmented object in the second selected region is segmented according to the pixel value derivation region, and the segmented object in the second selected region is obtained.

As described above, in the depth image, two different regions may be selected, and the division objects in the two regions may be divided by the depth value derivation method (step S11 to step S14) and the pixel value derivation method (step S15 to step S17). Then, the segmentation object in the depth image may also be determined from the segmentation objects in the two regions.

It is to be understood that steps S11 through S14 and steps S15 through S17 are shown in the order of fig. 9, but are not limited to this order, and steps S15 through S17 may be placed before steps S11 through S14.

Fig. 10 is a flowchart illustrating a method of segmentation of a depth image according to yet another exemplary embodiment. As shown in fig. 10, on the basis of the embodiment shown in fig. 9, the method may further include step S18.

In step S18, a segmentation object in the depth image is determined based on the segmentation object in the first selected region and the segmentation object in the second selected region.

Since the derivation method using depth values (step S11 to step S14) and the derivation method using pixel values (step S15 to step S17) each have their own advantages, the split objects of the two regions can be combined as appropriate to form the split object of the depth image.

Several specific embodiments combining the two methods for segmentation are described below.

1) In the case that the first selected area and the second selected area do not overlap with each other, step S18 may be: and merging the segmentation object in the first selected area and the segmentation object in the second selected area to form the segmentation object in the depth image.

In this embodiment, a segmentation object in the depth image may be divided into two different regions that do not overlap each other, and as the case may be, a segmentation method of depth values is applied in one region and a conventional method (segmentation method of pixel values) is applied in the other region. For example, a region with a large difference in depth between the background and the segmented object is selected as the first selected region, and a region with a large difference in color between the background and the segmented object is selected as the second selected region. After the segmentation objects in the two regions are obtained respectively, the segmentation objects of the two regions are merged together to form the segmentation region of the depth image.

FIG. 11 is a diagram illustrating the formation of a segmented object in a depth image according to an exemplary embodiment. As shown in fig. 11, the image of the person is divided in the depth image. The depth image may be divided into two regions, and the image of the person in the upper region is far from the background and is suitable for being divided by the method of steps S11 to S14, so that the upper region is selected as the first selected region. In the lower region, the image of the person is close to the background and the difference in color is large, and it is appropriate to divide it by the method of step S15 to step S17, and therefore the lower region is selected as the second selected region. After the segmentation is performed separately, the obtained segmentation objects in the two regions are combined to form a complete human image.

2) In the case that the first selected area and the second selected area overlap with each other to form an overlapping area, step S18 may be: merging a portion of the segmented object in the first selected region outside the overlapping region, a portion of the segmented object in the second selected region outside the overlapping region, and any one of:

the part of the segmentation object in the first selected area in the overlapping area; or

And the part of the segmentation object in the second selected area in the overlapping area.

That is, the part outside the overlap region may be retained, and the part inside the overlap region may be retained with a relatively good segmentation effect, and the retained regions may be combined to form a segmentation method of the depth image.

For example, after the segmentation object in the first selected region is segmented, if it is considered that which part is not segmented well, the second selected region may be re-selected from the depth image, the second selected region is re-segmented once by using the conventional segmentation method, and the re-segmentation result is used to replace the corresponding part in the first selected region. That is, the portion of the segmented object in the overlap region in the first selected region may be replaced with the portion of the segmented object in the second selected region in the overlap region.

Fig. 12 is a schematic diagram illustrating the formation of a segmented object in a depth image according to another exemplary embodiment. As shown in fig. 12, to segment the image of the person in the depth image on the left side of fig. 11, the whole image of fig. 11 may be selected as the first selected area, and the segmentation is performed by using the depth segmentation method first, so as to obtain the result on the left side of fig. 12. If it is observed that the segmentation result of the lower right corner is not ideal, the region in the dashed line frame of the lower right corner may be selected as the second selection region (at this time, the second selection region is included in the first selection region, and the second selection region is also an overlapping region), and the person image in the second selection region is segmented by using the conventional segmentation method. Then, the person image in the second selected region (i.e., the portion of the segmented object in the second selected region within the overlapping region) may be combined with the portion of the person image in the first selected region outside the second selected region (i.e., the portion of the segmented object in the first selected region outside the overlapping region) to form the entire person image (the segmented object of the depth image). It will be appreciated that there is no part of the segmented object in the second selected region outside the overlap region at this time.

The above-described embodiments all apply two segmentation methods (depth value segmentation method and conventional segmentation method), which can exert their advantages, and the segmentation effect is better than that of a better segmentation result with one method.

The present disclosure also provides a depth image segmentation apparatus. Fig. 13 is a block diagram illustrating a depth image segmentation apparatus according to an exemplary embodiment. As shown in fig. 13, the apparatus includes a depth value acquisition module 11, a depth value range estimation module 12, a depth value derivation area determination module 13, and a first segmentation module 14.

The depth value obtaining module 11 is configured to obtain depth values of pixel points in a first selected area in the depth image.

The depth value range estimation module 12 is configured to estimate a range of depth values of the segmented object in the first selected region.

The depth value derivation area determination module 13 is configured to determine pixel points in the first selected area whose depth values are within the depth value range, and form a depth value derivation area according to the determined pixel points.

The first segmentation module 14 is configured to segment the segmented object in the first selected area according to the depth value derivation area, resulting in the segmented object in the first selected area.

FIG. 14 is a block diagram illustrating the structure of depth value range estimation module 12 according to an exemplary embodiment. As shown in fig. 14, the depth value range estimation module 12 includes a distribution status determination unit 121 and a first depth value range estimation unit 122.

The distribution status determining unit 121 is configured to determine the distribution status of the depth values of all the pixels in the first selected region according to the obtained depth values of the pixels in the first selected region.

The first depth value range estimation unit 122 is configured to estimate a range of depth values of the segmented object in the first selected area in accordance with the determined distribution condition.

FIG. 15 is a block diagram illustrating the depth value range estimation module 12 according to another exemplary embodiment. As shown in fig. 15, the depth value range estimation module 12 includes a reference region selection unit 123, a reference depth value range determination unit 124, and a second depth value range estimation unit 125.

The reference region selection unit 123 is configured to select a reference region in the first selected region;

the reference depth value range determination unit 124 is configured to determine a depth value range of the reference area.

The second depth value range estimation unit 125 is configured to estimate a range of depth values of the segmented object in the first selected area from a range of depth values of the reference area.

Optionally, the segmentation object is a human image, and the reference region is a skin color region.

Fig. 16 is a block diagram illustrating a depth image segmentation apparatus according to another exemplary embodiment. As shown in fig. 16, on the basis of fig. 13, the apparatus further includes a pixel value acquisition module 15, a pixel value derivation area determination module 16, and a second division module 17.

The pixel value obtaining module 15 is configured to obtain pixel values of pixel points in a second selected region in the depth image.

The pixel value derivation area determination module 16 is configured to determine pixel points of the segmented object in the second selected area according to pixel values of the pixel points in the second selected area, and form a pixel value derivation area according to the determined pixel points.

The second segmentation module 17 is configured to segment the segmented object in the second selected area according to the pixel value derivation area, resulting in the segmented object in the second selected area.

Wherein the pixel value derivation area determination module 16 includes any one of:

a threshold derivation unit configured to determine pixel points of the segmented object in the second selected region according to a threshold-based segmentation method, and form a pixel value derivation region according to the determined pixel points.

An edge derivation unit configured to determine pixel points of the segmented object in the second selected region according to an edge-based segmentation method, and form a pixel value derivation region according to the determined pixel points.

And the region derivation unit is configured to determine pixel points of the segmentation object in the second selected region according to a region-based segmentation method, and form a pixel value derivation region according to the determined pixel points.

And the graph theory derivation unit is configured to determine pixel points of the segmentation object in the second selected region according to a graph theory-based segmentation method and form a pixel value derivation region according to the determined pixel points.

An energy functional derivation unit configured to determine pixel points of the segmented object in the second selected region according to an energy functional-based segmentation method, and form a pixel value derivation region according to the determined pixel points.

Fig. 17 is a block diagram illustrating a depth image segmentation apparatus according to another exemplary embodiment. As shown in fig. 17, on the basis of fig. 16, the apparatus further includes a segmented object determining module 18.

The object determination module 18 is configured to determine a segmentation object in the depth image from the segmentation object in the first selected region and the segmentation object in the second selected region.

Optionally, the first selection area and the second selection area are not overlapped with each other; and

the segmented object determination module 18 is configured to: merging the segmentation object in the first selected region and the segmentation object in the second selected region to form a segmentation object in the depth image.

Optionally, the first selection area and the second selection area are overlapped with each other to form an overlapped area; and

the segmented object determination module 18 is configured to: merging a portion of the segmented object in the first selected region outside the overlapping region, a portion of the segmented object in the second selected region outside the overlapping region, and any one of:

With regard to the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

According to the method and the device, the required object is segmented in the image according to the depth information acquired in the shooting process, large effective information blocks cannot be lost in the segmented image, large redundant information blocks cannot occur, and the edge of image segmentation is finer. Therefore, the depth image segmentation device provided by the disclosure can accurately segment a required segmentation object when the depth difference between the segmentation object and the background is large.

Fig. 18 is a block diagram illustrating a depth image segmentation apparatus 1800 according to an exemplary embodiment. For example, the apparatus 1800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.

Referring to fig. 18, the apparatus 1800 may include one or more of the following components: a processing component 1802, a memory 1804, a power component 1806, a multimedia component 1808, an audio component 1810, an input/output (I/O) interface 1812, a sensor component 1814, and a communications component 1816.

The processing component 1802 generally controls the overall operation of the device 1800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 1802 may include one or more processors 1820 to execute instructions to perform all or part of the steps of the depth image segmentation method described above. Further, the processing component 1802 may include one or more modules that facilitate interaction between the processing component 1802 and other components. For example, the processing component 1802 can include a multimedia module to facilitate interaction between the multimedia component 1808 and the processing component 1802.

The memory 1804 is configured to store various types of data to support operation at the apparatus 1800. Examples of such data include instructions for any application or method operating on the device 1800, contact data, phonebook data, messages, images, videos, and so forth. The memory 1804 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power components 1806 provide power to various components of the device 1800. The power components 1806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the apparatus 1800.

The multimedia component 1808 includes a screen providing an output interface between the apparatus 1800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1808 includes a front facing camera and/or a rear facing camera. The front-facing camera and/or the rear-facing camera may receive external multimedia data when the device 1800 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

Audio component 1810 is configured to output and/or input audio signals. For example, the audio component 1810 includes a Microphone (MIC) configured to receive external audio signals when the apparatus 1800 is in operating modes, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 1804 or transmitted via the communication component 1816. In some embodiments, audio component 1810 also includes a speaker for outputting audio signals.

I/O interface 1812 provides an interface between processing component 1802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor component 1814 includes one or more sensors for providing various aspects of state assessment for the apparatus 1800. For example, the sensor assembly 1814 can detect an open/closed state of the device 1800, the relative positioning of components, such as a display and keypad of the device 1800, the sensor assembly 1814 can also detect a change in position of the device 1800 or a component of the device 1800, the presence or absence of user contact with the device 1800, orientation or acceleration/deceleration of the device 1800, and a change in temperature of the device 1800. Sensor assembly 1814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 1816 is configured to facilitate communications between the apparatus 1800 and other devices in a wired or wireless manner. The device 1800 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1816 receives a broadcast signal or broadcast associated information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 1800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the depth image segmentation method described above.

In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as the memory 1804 including instructions executable by the processor 1820 of the apparatus 1800 to perform the depth image segmentation method described above. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of segmentation of a depth image, the method comprising:

obtaining the depth value of a pixel point in a first selected area in the depth image;

estimating a depth value range of the segmentation object in the first selected area;

determining pixel points of the first selected area, the depth values of which are within the depth value range, forming a depth value derivation area according to the determined pixel points, and displaying the depth value derivation area in the depth image so that a user can adjust the depth value range of the segmentation object in the first selected area according to the determined depth value derivation area; and

segmenting the segmentation object in the first selection area according to the depth value derivation area to obtain the segmentation object in the first selection area;

acquiring pixel values of pixel points in a second selected area in the depth image;

determining pixel points of the segmentation objects in the second selection area according to the pixel values of the pixel points in the second selection area, and forming a pixel value derivation area according to the determined pixel points; and

segmenting the segmentation object in the second selection area according to the pixel value derivation area to obtain the segmentation object in the second selection area;

and combining the segmentation object in the first selected area and the segmentation object in the second selected area to form the segmentation object in the depth image.

2. The method of claim 1, wherein the step of estimating the range of depth values for the segmented objects in the first selected region comprises:

determining the distribution condition of the depth values of all the pixel points in the first selected area according to the obtained depth values of the pixel points in the first selected area; and

and estimating the depth value range of the segmentation objects in the first selected area according to the determined distribution condition.

3. The method of claim 1, wherein the step of estimating the range of depth values for the segmented objects in the first selected region comprises:

selecting a reference area in the first selected area;

determining a range of depth values for the reference region; and

and estimating the depth value range of the segmentation object in the first selected area according to the depth value range of the reference area.

4. The method of claim 3, wherein the segmented object is a human image and the reference region is a skin tone region.

5. The method according to claim 1, wherein the step of determining pixel points of the segmented object in the second selected region according to pixel values of the pixel points in the second selected region and forming a pixel value derivation region according to the determined pixel points comprises any one of:

determining pixel points of the segmentation objects in the second selected region according to a threshold-based segmentation method, and forming a pixel value derivation region according to the determined pixel points;

determining pixel points of the segmentation object in the second selected region according to an edge-based segmentation method, and forming a pixel value derivation region according to the determined pixel points;

determining pixel points of the segmentation object in the second selected region according to a region-based segmentation method, and forming a pixel value derivation region according to the determined pixel points;

determining pixel points of the segmentation object in the second selected region according to a graph theory-based segmentation method, and forming a pixel value derivation region according to the determined pixel points; or

And determining pixel points of the segmentation object in the second selected region according to the segmentation method based on the energy functional, and forming a pixel value derivation region according to the determined pixel points.

6. The method of claim 1,

the first selection area and the second selection area do not overlap each other,

the step of determining the segmentation object in the depth image according to the segmentation object in the first selected region and the segmentation object in the second selected region is as follows: merging the segmented objects in the first selected region and the segmented objects in the second selected region to form segmented objects in the depth image,

alternatively, the first and second electrodes may be,

the first selection area and the second selection area are overlapped with each other to form an overlapped area,

the step of determining the segmentation object in the depth image according to the segmentation object in the first selected region and the segmentation object in the second selected region is as follows: merging a portion of the segmented object in the first selected region outside the overlapping region, a portion of the segmented object in the second selected region outside the overlapping region, and any one of:

7. An apparatus for segmenting a depth image, the apparatus comprising:

the depth value acquisition module is configured to acquire the depth value of a pixel point in a first selected area in the depth image;

a depth value range estimation module configured to estimate a depth value range of the segmented object in the first selected region;

a depth value derivation area determination module configured to determine pixel points in the first selected area having depth values within the depth value range, and form a depth value derivation area according to the determined pixel points, and display the depth value derivation area in the depth image, so that a user adjusts the depth value range of the segmentation object in the first selected area according to the determined depth value derivation area; and

a first segmentation module configured to segment the segmented object in the first selected area according to the depth value derivation area to obtain the segmented object in the first selected area;

the pixel value acquisition module is configured to acquire pixel values of pixel points in a second selected area in the depth image;

a pixel value derivation region determination module configured to determine pixel points of the segmented object in the second selection region according to pixel values of the pixel points in the second selection region, and form a pixel value derivation region according to the determined pixel points; and

a second segmentation module configured to segment the segmentation object in the second selected region according to the pixel value derivation region to obtain a segmentation object determination module of the segmentation object in the second selected region, and configured to merge the segmentation object in the first selected region and the segmentation object in the second selected region to form the segmentation object in the depth image.

8. The apparatus of claim 7, wherein the depth value range estimation module comprises:

a distribution status determining unit configured to determine a distribution status of the depth values of all the pixels in the first selected region according to the obtained depth values of the pixels in the first selected region; and

a first depth value range estimation unit configured to estimate a depth value range of the segmentation object in the first selected area according to the determined distribution condition.

9. The apparatus of claim 7, wherein the depth value range estimation module comprises:

a reference region selection unit configured to select a reference region in the first selected region;

a reference depth value range determination unit configured to determine a depth value range of the reference area; and

a second depth value range estimation unit configured to estimate a range of depth values of the segmented object in the first selected area from the range of depth values of the reference area.

10. The apparatus of claim 9, wherein the segmented object is a human image and the reference region is a skin tone region.

11. The apparatus of claim 7, wherein the pixel value derivation region determination module comprises any one of:

a threshold derivation unit configured to determine pixel points of the division object in the second selected region according to a threshold-based division method, and form a pixel value derivation region according to the determined pixel points;

an edge derivation unit configured to determine pixel points of the division object in the second selected region according to an edge-based division method, and form a pixel value derivation region according to the determined pixel points;

a region derivation unit configured to determine pixel points of the division object in the second selected region according to a region-based division method, and form a pixel value derivation region according to the determined pixel points;

a graph theory derivation unit configured to determine pixel points of the division object in the second selected region according to a graph theory-based division method, and form a pixel value derivation region according to the determined pixel points; or

12. The apparatus of claim 7,

the first selection area and the second selection area are not overlapped with each other; and

the segmented object determination module is configured to: merging the segmented objects in the first selected region and the segmented objects in the second selected region to form segmented objects in the depth image,

alternatively, the first and second electrodes may be,

the segmented object determination module is configured to: merging a portion of the segmented object in the first selected region outside the overlapping region, a portion of the segmented object in the second selected region outside the overlapping region, and any one of:

13. A depth image segmentation apparatus, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: