JP6787844B2 - Object extractor and its superpixel labeling method - Google Patents

Description

The present invention relates to an object extraction device that extracts an object from a camera image and a superpixel labeling method thereof, and in particular, superpixels can be labeled with high accuracy with a small processing load, and based on this labeling result, the superpixels can be labeled from the camera image. The present invention relates to an object extraction device capable of accurately extracting an object and a superpixel labeling method thereof.

In Patent Document 1, the input camera image is divided into a large number of superpixels, and each superpixel in the image is classified into either "object" or "background" and labeled based on the corresponding object mask. However, a technique for extracting an object based on the labeling result is disclosed.

Non-Patent Document 1 discloses a 3D visual intersection-based model for labeling each 3D voxel as an "object" or "background" by comparing the projection of the 3D voxel to a plurality of calibrated cameras. Has been done. Non-Patent Document 2 discloses an energy optimization method based on graph theory.

Japanese Patent Application No. 2017-97953

Hiroshi Sankoh, Mehrdad Panahpour Tehrani, Sei Naito, "Method and program for extracting silhouette image and method and program for constructing three dimensional model", US 8363941 B2 Rother, C., Kolmogorov, V., & Blake, A. (2004, August). Grabcut: Interactive foreground extraction using iterated graph cuts. In ACM transactions on graphics (TOG) (Vol. 23, No. 3, pp. 309-314). ACM.

Patent Document 1 does not consider the treatment of "unknown" which cannot be clearly classified into either "object" or "background".

Non-Patent Document 1 is very sensitive because 3D voxel labeling is performed at the pixel or voxel level, and missing areas and discontinuous areas may occur in the extracted mask. Therefore, it was not possible to avoid the trade-off between the removal of unnecessary pixels and the restoration of missing pixels.

Non-Patent Document 2 requires allocation of an initial object area and a background area, and although there is no problem if a single object is extracted, a satisfactory result cannot be obtained when there are a plurality of objects. Further, if the color distribution inside the object and the color distribution of the background are similar, the texture inside the object is regarded as the background and easily removed, and the extracted mask area is lost.

An object of the present invention is an object extraction device that solves the above technical problems, can classify and label superpixels with high accuracy with a small processing load, and can accurately extract objects from a camera image based on the labeling result. And its superpixel labeling method.

In order to achieve the above object, the present invention presents a means for obtaining an object mask M ^k of an image, a means for dividing an image into a plurality of super pixels, and each super in an object extraction device for extracting an object from an image. The first classification method that classifies pixels into either "object", "background", or "unknown" by the first classification method based on the object mask M ^k , and each superpixel classified as "unknown" is "object". An unknown classification means for classifying into "" or "background" and a means for reviewing the object mask M ^k based on each classification result are provided, and objects are extracted based on the revised object mask M ^k .

According to the present invention, the following effects are achieved.

(1) Since other classification methods are additionally applied only to superpixels classified into the category "Unknown" by the first classification method, the selectivity is high for all superpixels, in other words. Compared with the case of applying a classification method with a large calculation load, the processing load related to category classification can be reduced.

(2) For super pixels classified into the category "Unknown" by the first classification method, a classification method with high selectivity for "mask accuracy" and "super pixel division error", which are likely to be unknown causes. Is applied, so highly accurate classification and labeling are possible.

(3) Based on the histogram peculiar to "object" and the histogram peculiar to "background", the probability that the feature vector of the superpixel classified as "unknown" is "object" and the probability that it is "background" is calculated. Since classification and labeling are performed, adaptive and highly accurate classification that does not depend on a quantitative threshold value becomes possible.

It is a block diagram of the object extraction apparatus to which this invention is applied. It is a figure which showed an example of a camera image. It is a figure which showed an example of the object mask M ^k . It is a functional block diagram which showed the structure of the object mask extraction apparatus which concerns on 1st Embodiment of this invention. It is a figure which showed an example of the classification result by the 1st classification part (40). It is a figure which showed each classification result of "object", "background" and "unknown" by comparison with a camera image. It is a figure which showed the subdivision method of a super pixel. It is a flowchart which showed the classification procedure of super pixel pi by the 1st classification method. It is a flowchart which showed the classification procedure of super pixel pi by the 2nd classification method. It is a flowchart which showed the classification procedure of super pixel pi by the 3rd classification method. It is a functional block diagram which showed the structure of the object mask extraction apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which showed the classification procedure of super pixel pi by the 4th classification method. It is a functional block diagram which showed the structure of the object mask extraction apparatus which concerns on 3rd Embodiment of this invention. It is a functional block diagram which showed the structure of the object mask extraction apparatus which concerns on 4th Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing an outline of an object extraction device to which the present invention is applied. The object extraction device of the present invention can be configured by implementing an application (program) that realizes each function on a general-purpose computer or server. Alternatively, a part of the application may be configured as a dedicated machine or a single-purpose machine that is made into hardware or ROM.

The multi-viewpoint image input unit 10 determines a predetermined camera image (including a moving image) shown in FIG. 2 from a plurality of cameras Cak (Ca1, Ca2 ...) That shoot the object Obj from different viewpoints for each viewpoint. Get in a cycle. The object mask generation unit 20 obtains a difference image by subtracting the background image taken in an environment where no object exists from the camera image, and compares each pixel of the difference image with a predetermined threshold value to obtain a difference image. Generate an object mask M ^k showing an example.

The super pixel division unit 30 divides the camera image into a large number of super pixels p1, p2 ... pk (hereinafter, represented by pi). The super pixel pi is a perceptually meaningful small area in which a plurality of pixels having relatively similar properties such as brightness and color are grouped together. As a method for calculating superpixels, a Graph-based method, a Mean Shift method, a Normalized Cut method, and the like are known. Each superpixel pi has useful boundary information for defining the object mask.

As will be described in detail later, the first classification unit 40 obtains the ratio (occupancy rate ρ _i ) of the object area to each super pixel pi based on the object mask M _k, and sets each super pixel pi to the occupancy rate ρ _i . Based on this, the first classification method of classifying into one of the categories of "object (class 1)", "background (class 2)" and "unknown (class 3)" is adopted.

The unknown classification unit 100 applies one or more classification methods different from the first classification method to only each superpixel classified as "unknown" by the first classification unit 40, and uses them together to form an "object" or Classify as "background". In the present embodiment, the unknown classification unit 100 has a second classification unit 50, a third classification unit 60, and / or a fourth classification unit 80, which will be described later.

The object mask review unit 70 reviews the object mask M ^k based on the classification result. In the present embodiment, the pixel range corresponding to each pixel in the super pixel pi with the "object" label L _obj is modified to the object area, and each pixel in the super pixel pi with the background label L _bg is modified. The corresponding pixel range is modified to the background area.

FIG. 4 is a functional block diagram showing the configuration of the object mask extraction device according to the first embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. In the present embodiment, the unknown classification unit 100 includes a second classification unit 50 and a third classification unit 60.

In the first classification unit 40, the object occupancy rate calculation unit 401 sets each super pixel pi based on whether or not the pixel of the object mask M ^k corresponding to each pixel in the super pixel pi is an object area. Calculate the object occupancy rate ρ _i , which indicates the ratio of the object area to the super pixel pi.

The object identification unit 402 compares the object occupancy rate ρ _i with the predetermined object identification threshold ρ _{th1_obj} for each super pixel pi, and classifies the super pixel pi into the category “object” when the following equation (1) is satisfied. ..

ρ _{th1_obj} <ρ _i <1… (1)

The background identification unit 403 compares the object occupancy rate ρ _i with the predetermined background identification threshold value ρ _{th1_bg} for each super pixel pi, and classifies the super pixel pi into the category “background” when the following equation (2) is satisfied. ..

0 <ρ _i <ρ _{th1_bg} … (2)

The labeling unit 404 attaches the "object" label L _obj to the super pixel pi of the category "object" and attaches the background label L _bg to the super pixel pi of the category "background". Note that superpixel pi that is not classified in either the "object" or "background" category is classified in the category "unknown" and is not labeled at this point.

FIG. 5 is a diagram showing an example of the classification result by the first classification unit 40, in which the white portion (pixel value = 255) is classified into the category “object” and the black portion (pixel value = 0). Areas classified into the category "Background". In addition, as shown in a partially enlarged view, the gray part mainly located at the boundary between the "object" area and the "background" area is not classified into either "object" or "background". This is the area of "unknown".

FIG. 6 is a diagram showing each classification result of “object”, “background”, and “unknown” by comparison with a camera image. According to the inventors of the present invention, the reason why a part of the super pixel pi is classified into the category "unknown" is due to a mistake in superpixel division as in the category "unknown (1)" and the category "unknown (unknown). 2) ”, we newly discovered that it can be roughly divided into those based on the accuracy of the object mask.

Therefore, in the present invention, the second classification method that enables highly selective classification with high selectivity regardless of the mask accuracy, and the selectivity regardless of the mistake of superpixel division, only for the region of the category "unknown". By applying the third classification method that enables high-precision classification of the "unknown" area, the "unknown" area is classified into either "object" or "background".

Returning to FIG. 4, the second classification unit 50 targets only the super pixel pi of the category “unknown”, and by applying a suitable second classification method of the classification of “unknown (2)”, the “object” or Classify into one of the "background" categories. Then, the object label L _obj is attached to the super pixel pi classified into the category "object", and the background label L _bg is attached to the super pixel classified into the category "background".

In the second classification unit 50, the object mask correction unit 501 applies to the object masks M ^k-1 and M ^{k + 1} of a plurality of other camera images having different viewpoints from the object mask M ^k , as will be described in detail later. Based on this, the object mask M ^k is modified.

The classification retry unit 502 classifies each super pixel pi of the category "unknown" into any of the categories of "object", "background", and "unknown" based on the modified object mask M ^k' . The labeling unit 503 attaches the object label L _obj to the super pixel pi of the category "object" and attaches the background label L _bg to the super pixel pi of the category "background".

The third classification unit 60 applies a third classification method particularly suitable for the classification of "unknown (1)" to the super pixel pi of the category "unknown" even after the classification by the second classification unit 50. To classify into either "object" or "background" category. Then, the object label L _obj is attached to the super pixel of the category "object", and the background label L _bg is attached to the super pixel of the category "background".

In the third classification unit 60, the super pixel subdivision unit 601 further divides the super pixel pi of the category “unknown (1)” into a plurality of super pixels, as shown in FIG. The subdivision unit 602 classifies each superpixel pi into an "object" or a "background" based on its object occupancy rate ρ _i .

Then, the superpixel division is repeated for the superpixels that are still "unknown", and finally all the superpixels are classified into "object" or "background". The labeling unit 603 attaches the object label L _obj to the super pixel pi of the category "object" and attaches the background label L _bg to the super pixel pi of the category "background".

FIG. 8 is a flowchart showing the classification procedure of the super pixel pi by the first classification unit 40.

In step S41, the super pixel pi of the camera image is acquired from the super pixel dividing unit 30. In step S42, the object mask M ^k is acquired from the object mask generation unit 20. In step S43, the object occupancy rate calculation unit 401 calculates the object occupancy rate ρ _i for each super pixel pi based on the object mask M ^k .

In step S44, the object identification unit 402 pays attention to one of the super pixel _pis, and _compares the object identification threshold value ρt _{h1_obj} with the object occupancy rate ρ _i to determine whether or not the attention _superpixel pi this time is an object area. Make a decision based on the results. In the present embodiment, if the above equation (1) is satisfied, the area is identified as an object area and the process proceeds to step S45, and the superpixel pi of interest this time is classified into the category “object”.

On the other hand, if the above equation (1) is not satisfied, the process proceeds to step S46, and the background identification unit 403 determines whether or not the _superpixel pi of interest this time is the background area with the background identification threshold value ρ _{th1_bg} and the object occupancy. Judgment is made based on the comparison result with the rate ρ _i . In the present embodiment, if the above equation (2) is satisfied, the area is identified as the background area and the process proceeds to step S47, and the superpixel pi of interest this time is classified into the category “background”.

In step S48, the labeling unit 404 _{assigns the} object label L _obj to the superpixels of the category “object” and the background label L _bg to the superpixels of the category “background”. Superpixels that are not classified as either "object" or "background" are classified in the category "unknown" and are not labeled.

In step S49, it is determined whether or not the above classification process is completed for all the super pixel pi. If it is not completed, the process returns to step S44, and each of the above processes is repeated while switching the super pixel pi of interest.

FIG. 9 is a flowchart showing a classification procedure for the super pixel pi of the category “unknown” by the second classification unit 50.

In step S51, superpixel pi of the category "unknown" that is not classified into either "object" or "background" is acquired. In step S52, a plurality of object masks M ^k-1 , M ^{k + 1,} and their camera parameters generated for other camera images are acquired.

In step S53, a 3D visual hull is constructed based on the object mask M ^k and the object masks M ^k-1 and M ^{k + 1} for each reference, and each coordinate of the ^superpixel of category "unknown" in the mask M ^k is set. Corresponding pixels are identified by projecting onto the reference object masks M ^k-1 and M ^{k + 1} . In step S54, the object mask correction unit 501 corrects the mask M ^k in comparison with the corresponding pixel.

In the present embodiment, the pixel value at the coordinates x, y of the object mask M ^k is M ^k (x, y), and the pixel value at the corresponding coordinates x ^k-1 and y ^k-1 of the object mask M ^k-1 is M ^{k. -1} (x ^k-1 , y ^k-1 ), the pixel value at the corresponding coordinates x ^{k + 1} , y ^{k + 1} of the object mask M ^{k + 1} is M ^{k + 1} (x ^{k + 1} , y ^{k + 1)} ), The object mask M ^k is modified based on the following equation.

That is, for the coordinates where the pixel values of the reference object masks M ^k-1 and M ^{k + 1} are both 0 (background), the pixel value is "0" regardless of the current pixel value of the object mask M ^k. Is rewritten as. Similarly, for the coordinates where the pixel values of the reference object masks M ^k-1 and M ^{k + 1} are 255 (objects), the pixel values are "255" regardless of the current pixel values of the object mask M ^k. Is rewritten as. If the pixel values of the reference object masks M ^k-1 and M ^{k + 1} do not match, the current pixel values of the object mask M ^k are maintained.

In step S55, the object occupancy rate ρ _i is recalculated for each superpixel pi of the category “unknown” based on the modified object mask M _k '.

In step S56, paying attention to one of the super pixel pi, whether or not the super pixel pi of interest this time corresponds to the object area is based on the comparison result between the object determination threshold value ρ _{th2_obj} and the object occupancy rate ρ _i. Is judged. In the present embodiment, if the following equation (4) is satisfied, the area is identified as an object area and the process proceeds to step S57, and the superpixel pi of interest this time is classified into the category “object”.

ρ _{th2_obj} <ρ _i <1… (4)

On the other hand, if the above equation (4) does not hold, the process proceeds to step S58, and whether or not the _superpixel pi of interest this time corresponds to the background area is determined by the background determination threshold value ρ _{th2_bg} and the object occupancy rate ρ _i . Judgment is made based on the comparison result of. In the present embodiment, if the following equation (5) is satisfied, the area is identified as the background area and the process proceeds to step S59, and the superpixel pi of interest this time is classified into the category “background”.

0 <ρ _i <ρ _{th2_bg} … (5)

In step S60, the labeling unit 503 _{assigns the} object label L _obj to the superpixels of the category “object” and the background label L _bg to the _{superpixels of} the category “background”. Superpixels that are not classified as either "object" or "background" remain in the category "unknown" and are not labeled.

In step S61, it is determined whether or not the above classification process has been completed for all superpixels in the category "unknown". If it is not completed, the process returns to step S56, and each of the above processes is repeated while switching the "unknown" superpixel pi of interest.

FIG. 10 is a flowchart showing a classification procedure for super pixels of the category “unknown” by the third classification unit 60.

In step S70, the "unknown" superpixel pi is still acquired. In step S71, as shown in FIG. 7A, an circumscribed rectangle is set in the superpixel pi. In step S72, as shown in FIG. 7B, the inside of the circumscribing rectangle is divided into a plurality of super pixels.

In step S73, the object occupancy rate ρ _i is calculated for each superpixel pi. In step S74, each superpixel pi is classified into one of "object", "background", and "unknown" based on the relationship between the object occupancy rate ρ _i and the object identification threshold value and the background identification threshold value, as described above. And labeled.

In step S75, it is determined whether the "unknown" superpixel pi is sufficiently low. If the number is not sufficiently small, the process returns to step S71, and the previously "unknown" superpixel pi is further decomposed into superpixels, and each of the above processes is repeated.

On the other hand, when the number of "unknown" superpixel pi is sufficiently small, the process proceeds to step S76, and the labeling unit 603 _{attaches the} object label L _obj to the superpixel of the category "object" and _supers of the category "background". Pixels have a background label L _bg . Superpixels that are not classified as either "object" or "background" remain in the category "unknown" and are not labeled.

In step S77, it is determined whether or not the above processing is completed for all the "unknown" super pixel pi extracted in step S70. If it is not completed, the process returns to step S70, and each of the above processes is repeated while switching the "unknown" superpixel pi of interest.

According to the present embodiment, since another classification method is additionally applied only to the super pixels classified into the category "Unknown" by the first classification method, the selectivity is high for all super pixels. In other words, the processing load related to category classification can be reduced as compared with the case of applying a classification method having a large calculation load.

Further, according to the present embodiment, for superpixels classified into the category "unknown" by the first classification method, for "mask accuracy" and "superpixel division error" which are likely to be unknown causes. Since a highly selective classification method is applied, highly accurate classification and labeling are possible.

FIG. 11 is a functional block diagram showing the configuration of the object mask extraction device according to the second embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. In the present embodiment, the unknown classification unit 100 is composed of the fourth classification unit 80, and the fourth classification unit 80 reclassifies the super pixel pi determined by the first classification unit 40 to be “unknown”. There is a feature in.

In the fourth classification unit 80, the feature vector generation unit 801 constructs the feature vector for all the super pixel pi. In this embodiment, the superpixel pi is converted into the Lab color space, and the average values of all the pixels of the superpixel pi in the Lab color space μ _l , μ _a , μ _b and the covariance of the color components in the Lab color space ρ _ll. , Ρ _aa , ρ _bb , ρ _la , ρ _lb , ρ _ab , and the vectors Vpi = {μ _l , μ _a , μ _b , ρ _ll , ρ _aa , ρ _bb , ρ _la , ρ _lb , ρ _ab } is constructed.

The object histogram generation unit 802 quantitatively shows the distribution of each feature amount by individually voting the feature amount of each feature vector for all the superpixel pi classified into the category “object” Hist _obj. To generate.

The background histogram generation unit 803 quantitatively shows the distribution of each feature amount by individually voting the feature amount of each feature vector for all the superpixel pi classified in the category “background” Hist _bg. To generate.

The probability calculation unit 804 compares the feature vector of each superpixel pi of the category "unknown" with the object histogram Hist _obj and the background histogram Hist _bg , so that the probability of being an object for each superpixel pi of the category "unknown". Calculate R _obj and the probability R _bg that is the background.

The determination unit 805 classifies the superpixel pi with probability R _obj > probability R _{bg into} the category "object", and the superpixel pi with probability R _obj ≤ probability R _{bg into} the category "background". The labeling unit 806 attaches the "object" label L _obj to the super pixel pi of the category "object" and attaches the background label L _bg to the super pixel pi of the category "background".

FIG. 12 is a flowchart showing the classification procedure by the fourth classification unit 80, and in step S81, super pixel pi classified into each category of “object”, “background”, and “unknown” is acquired. In step S82, the feature vector generation unit 801 generates the feature vector Vpi for all the superpixel pi.

In this embodiment, the average values of all pixels of the superpixel pi in the Lab color space μ _l , μ _a , μ _b and the covariance of the color components in the Lab color space ρ _ll , ρ _aa , ρ _bb , ρ _la , ρ A feature vector Vpi = {μ _l , μ _a , μ _b , ρ _ll , ρ _aa , ρ _bb , ρ _la , ρ _lb , ρ _ab } with _lb and ρ _ab as feature quantities is generated.

In step S83, the object histogram generation unit 802 generates the object histogram Hist _obj by individually voting each feature amount of the feature vector for all the super pixel pi classified into the category “object”. Similarly, the background histogram generation unit 803 generates the background histogram Hist _bg by individually voting each feature amount of the feature vector for all the super pixel pi classified in the category “background”.

In step S84, the probability calculation unit 804 compares the feature vector of each superpixel pi of the category “unknown” with the object histogram Hist _obj and the background histogram Hist _bg , so that each superpixel pi of the category “unknown” is an object. Calculate the probability R _obj that is and the probability R _bg that is the background.

In step S85, the magnitude relation between the probability R _obj and the probability R _bg is compared, and if R _obj > R _bg , the process proceeds to step S86 and the object is classified into the category “object”. On the other hand, if R _obj ≤ R _bg , the process proceeds to step S87 and is classified into the category “background”. In step S88, the labeling unit 806 attaches the "object" label L _obj to the super pixel pi of the category "object" and attaches the background label L _bg to the super pixel pi of the category "background".

According to the present embodiment, based on the histogram specific to "object" and the histogram specific to "background", the probability that the feature vector of the superpixel classified as "unknown" is "object" and the "background" Since classification and labeling are performed by finding a certain probability, adaptive and highly accurate classification that does not depend on a quantitative threshold is possible.

FIG. 13 is a functional block diagram showing the configuration of the object mask extraction device according to the third embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. In this embodiment, the unknown classification unit 100 includes a second classification unit 50 and a fourth classification unit 80, and the second classification unit 50 has a second classification unit 50 for a super pixel classified into the category “unknown” by the first classification unit 40. The feature is that the second classification unit 50 performs reclassification, and the fourth classification unit 80 further reclassifies the super pixels classified into the category "unknown".

FIG. 14 is a functional block diagram showing the configuration of the object mask extraction device according to the fourth embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. In the present embodiment, the unknown classification unit 100 is composed of the third classification unit 60, and the third classification unit 60 classifies the super pixels classified into the category "unknown" by the first classification unit 40. There is a feature in.

10 ... Multi-view image input unit, 20 ... Object mask generation unit, 30 ... Super pixel division unit, 40 ... 1st classification unit, 50 ... 2nd classification unit, 60 ... 3rd classification unit, 70 ... Object mask review unit, 80 ... 4th classification unit, 401 ... object occupancy calculation unit, 402 ... object identification unit, 403 ... background identification unit, 404 ... labeling unit, 501 ... object mask correction unit, 502 ... classification retry unit, 503 ... labeling unit , 601 ... Super pixel subdivision unit, 602 ... Subdivision unit, 603 ... Labeling unit, 801 ... Feature vector generation unit, 802 ... Object histogram generation unit, 803 ... Background histogram generation unit, 804 ... Probability calculation unit, 805 ... Judgment Part, 806 ... Labeling part

Claims (14)

In an object extraction device that extracts objects from images including moving images
Means (20) for obtaining the object mask Mk of the image and
A first classification means (40) that classifies each superpixel of the image into one of the first, second, and third categories by the first classification method based on the object mask Mk.
An unknown classification means (100) that further classifies each superpixel classified into the third category by a classification method different from the first classification method.
An object extraction device provided with a means (70) for reviewing by modifying the object area and the background area of the object mask Mk based on the results of each of the classifications. The unknown classification means includes a second classification means (50) for classifying the superpixels classified into the third category into any of the first, second and third categories.
The second classification means (50) is
A means (501) for modifying the object mask Mk based on the object masks Mk-1 and Mk + 1 of a plurality of images having different viewpoints from the image, and
The claim is characterized in that the superpixel of the third category is provided with a means (502) for classifying the super pixel of the third category into any of the first category, the second category and the third category based on the modified object mask Mk'. The object extraction device according to 1. The unknown classification means includes a third classification means (60) for classifying the superpixels classified into the third category into the first or second category.
The third classification means (60) is
A means of subdividing a third-class superpixel into multiple superpixels (601), and
It is provided with a reclassification means (602) for reclassifying each subdivided superpixel into any of Class 1, Class 2, and Class 3 based on the object mask Mk.
The object extraction device according to claim 1 , wherein the superpixels of the third category are repeatedly subdivided and reclassified even after the reclassification. The unknown classification means includes a third classification means (60) for classifying superpixels classified into the third category by the second classification means into the first category or the second category .
The third classification means (60) is
A means of subdividing a third-class superpixel into multiple superpixels (601), and
It is provided with a reclassification means (602) for reclassifying each subdivided superpixel into any of Class 1, Class 2, and Class 3 based on the object mask Mk.
The object extraction device according to claim 2 , wherein the superpixels of the third category are repeatedly subdivided and reclassified even after the reclassification . The unknown classification means includes a fourth classification means (80) for classifying superpixels classified into the third category into the first or second category.
The fourth classification means (80) is
A means to generate a feature vector for each superpixel (801) and
A means (802) to generate an object histogram unique to the first category based on the feature vector of each superpixel classified into the first category.
A means (803) for generating a background histogram unique to the second category based on the feature vector of each superpixel classified into the second category, and
A means (804) for calculating the probability that each superpixel is classified into the first category and the probability that each superpixel is classified into the second category by comparing the feature vector of each superpixel classified into the third category with each histogram.
The claim is characterized by comprising a determination means (805) for determining whether each superpixel classified into the third category is a first category or a second category based on the respective probabilities. The object extraction device according to 1. The unknown classification means includes a fourth classification means (80) for classifying superpixels classified into the third category by the second classification means into the first category or the second category.
The fourth classification means (80) is
A means to generate a feature vector for each superpixel (801) and
A means (802) to generate an object histogram unique to the first category based on the feature vector of each superpixel classified into the first category.
A means (803) for generating a background histogram unique to the second category based on the feature vector of each superpixel classified into the second category, and
A means (804) for calculating the probability that each superpixel is classified into the first category and the probability that each superpixel is classified into the second category by comparing the feature vector of each superpixel classified into the third category with each histogram.
The claim is characterized by comprising a determination means (805) for determining whether each superpixel classified into the third category is a first category or a second category based on the respective probabilities. 2. The object extraction device according to 2. The first classification means (40) is
A means of calculating the object occupancy for each superpixel based on the object mask Mk (401) and
A means (402) for classifying superpixels whose object occupancy exceeds a predetermined object threshold into the first category, and
It is provided with a means (403) for classifying superpixels whose object occupancy rate is below a predetermined background threshold into the second category.
The object extraction device according to any one of claims 1 to 6, wherein a super pixel that is not classified into any of the first and second categories is classified into the third category. The object extraction device according to any one of claims 1 to 7 , wherein an object is extracted based on the revised object mask Mk. The object extraction device according to any one of claims 1 to 8 , wherein the first class, the second class, and the third class are "object", "background", and "unknown", respectively. In the super pixel labeling method of the object extractor,
Get the object mask Mk of the image including the moving image,
Divide the image into multiple superpixels
Each superpixel is classified into one of the first, second and third categories by the first classification method based on the object mask Mk.
Each superpixel classified into the third category is classified into the first category or the second category by a classification method different from the first classification method.
A superpixel labeling method for an object extractor, characterized in that each superpixel is labeled based on the results of each of the classifications. A classification method different from the first classification method includes a second classification method.
The second classification method is
The object mask Mk is modified based on the object masks Mk-1 and Mk + 1 of a plurality of images having different viewpoints from the image.
The object extraction device according to claim 10 , wherein a third-class superpixel is classified into one of the first, second, and third types based on the modified object mask Mk'. Super pixel labeling method. A classification method different from the first classification method includes a third classification method.
The third classification method is
Subdivide the third class superpixel into multiple superpixels,
Reclassify each subdivided superpixel into one of Class 1, Class 2, and Class 3 based on the object mask Mk.
The superpixel labeling method for an object extraction device according to claim 10 , wherein the subdivision and reclassification of the third type of superpixels are repeated even after the reclassification. A classification method different from the first classification method includes a fourth classification method.
The fourth classification method is step (80).
Generate a feature vector for each superpixel
Generate an object histogram unique to the first category based on the feature vector of each superpixel classified in the first category.
A background histogram unique to the second category is generated based on the feature vector of each superpixel classified into the second category.
The feature vector of each superpixel classified into the third category is compared with each histogram to calculate the probability that each superpixel is the first category and the second category.
The supermarket of the object extraction apparatus according to claim 10 , wherein it is determined whether each superpixel classified into the third category is classified into the first category or the second category based on the respective probabilities. Pixel labeling method. The superpixel labeling method according to any one of claims 10 to 13 , wherein the first class, the second class, and the third class are "object", "background", and "unknown", respectively.