JP2020119250A - Object extraction method and device - Google Patents

Abstract

To provide an object extraction method and device for extracting an object by separating foreground from a moving image.SOLUTION: An object region detection part 2 detects an object region for each frame from a camera video acquired by a camera video acquisition part 1. A presence likelihood map calculation part 3 calculates a presence likelihood map indicating presence likelihood of an object for each frame on the basis of a result of the detection of the object region. A statistical information calculation part 6 calculates statistical information (an average and a standard deviation) of each pixel for each frame. A background difference calculation part 7 extracts, as the object, a foreground region by a background difference method based on the statistical information and a background difference threshold value. The statistical information calculation part 6 updates past statistical information of each pixel at a predetermined update ratio on the basis of a current pixel value to thereby obtain current statistical information. Further included is an update ratio determination part 5 which determines the update ratio on the basis of the presence likelihood map.SELECTED DRAWING: Figure 1

Description

The present invention relates to an object extraction method and device, and more particularly to an object extraction method and device that extracts an object by separating a foreground from a moving image.

Many methods have been proposed for separating a foreground region and a background region, mainly for the purpose of detecting a moving object in an image and producing a free-viewpoint image. In particular, an approach in which a background region in an image is modeled based on statistical information or the like and a region having a large difference between the background model and an input image is extracted as a foreground is called a background subtraction method.

As an example of the background subtraction method, Non-Patent Document 1 specifies the background region of the input image and extracts only the foreground by modeling the background using a mixed Gaussian distribution obtained by mixing a plurality of Gaussian distributions. Techniques for doing so are disclosed.

Non-Patent Document 2 discloses a method of modeling the background with a single Gaussian distribution and calculating the background difference while updating the background statistical information represented by the mean and the variance in each frame. In this method, after extracting candidate regions of the foreground based on the Gaussian distribution, the candidate regions should be reclassified into the foreground and the background based on the shape and histogram of the candidate regions, which should not be the original foreground such as shadows. It is possible to eliminate the parts that do not exist.

Non-Patent Document 3 discloses a method of increasing the accuracy of the background difference by adaptively changing the threshold for extracting the foreground set when performing the background difference based on the tracking information of the person. There is.

On the other hand, in recent years, a technique for extracting the silhouette of the target object using deep learning as represented by Non-Patent Document 4 has also been proposed. This method is a technique that prepares training data in advance and can extract the silhouette of an object based on prior learning using a convolutional neural network. Since this method extracts the target object based on the training data, when this method is applied to the separation of the foreground and the background, shadows are difficult to extract, and the target object is robust against changes in lighting conditions. It had the feature that it could be extracted.

C. Stauffer and W.E.L.Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252 Vol. 2 (1999). Q. Yao, H. Sankoh, H. Sabirin and S. Naito, "Accurate silhouette extraction of multiple moving objects for free viewpoint sports video synthesis," 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP), 2015, pp. 1 -6 (2015). Kenji Terabayashi, Kazunobu Umeda, Moro Alessandro, "Real-time adaptive threshold processing of background difference using human tracking information", Institute of Electrical Engineers of Japan, GID-09-17, pp.89-90 (2009). K. He, G. Gkioxari, P. Dollar and R. Girshick, "Mask R-CNN," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980-2988 (2017). H. Sankoh, S. Naito, K. Nonaka, H. Sabirin, J. Chen, "Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes", Proceedings of the 26th ACM international conference on Multimedia, pp. 1724-1732(2018) Z. Cao, T. Simon, S. Wei and Y. Sheikh, "Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302-1310 (2017 ). J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6517-6525 (2017).

The inventors of the present invention use the silhouette extracted by the background difference method-based method disclosed in Non-Patent Documents 1 and 2 to create a free-viewpoint video represented by Non-Patent Document 5. I went. In the production process of the free-viewpoint video technology disclosed in Non-Patent Document 5, a silhouette image is created using the background subtraction method, and then a visual intersection is calculated from each silhouette by calculating a product set in a three-dimensional space. Generate and 3D model the target person. The accuracy of silhouette extraction at this time greatly affects the quality of the free-viewpoint image.

The methods disclosed in Non-Patent Documents 1 and 2 statistically model the background and update the background. Therefore, robust target object extraction is performed for moderate background changes and regular background part changes. It had the advantage of being able to do it. However, it was difficult to apply it to a scene where the background is complicated and changes rapidly. As an example of a scene with a complicated background described here, for example, when a player or a ball is to be extracted in a sports game, a liquid crystal display for advertisement display is arranged behind the player, which frequently switches. The scene and the scene where the white line on the field that was drawn in baseball and the like is trampled along with the runner of the player are applicable.

In such a scene, in addition to the sudden change in the background, there is no regularity in the change in the background, so there is a high possibility that the background will be erroneously extracted as the foreground. In order to perform extraction in these difficult scenes, the methods of Non-Patent Documents 1 and 2 are insufficient in accuracy.

For such a technical problem, as in Non-Patent Document 3, a method has been proposed in which person tracking is performed and the result is applied to a threshold of the background subtraction method. Since Non-Patent Document 3 tracks a person and adjusts the threshold value based on the result, it is characterized by being robust against changes in illumination and the like as compared with Non-Patent Documents 1 and 2.

However, even if the threshold for discriminating between the foreground and the background is dynamically adjusted by the method of Non-Patent Document 3, when the background model is statistically updated, a player who has been stationary for a long time When it exists, there is a problem in that the player part that is gradually stationary is determined as the background model, and the target object is determined as the background. On the other hand, if there is no mechanism for statistically updating the background, the robustness against variations in illumination will be lost.

On the other hand, the deep learning-based method as in Non-Patent Document 4 has the advantage that it is robust against changes in lighting because the target object is detected from the feature amount of the entire image, and shadows and the like are difficult to extract as the foreground. It was

However, when occlusion occurs due to overlapping of extracted objects, there are many recognition omissions, and it is difficult to remove the contours neatly, and the contours of the target object are accurately extracted with the silhouette used for free-viewpoint video production. In view of the need to do so, the method of Non-Patent Document 4 was difficult to apply.

An object of the present invention is to solve the above technical problems and to provide an object extraction method and apparatus capable of accurately extracting an object having a long stationary time.

In order to achieve the above-mentioned object, the present invention is characterized in that an object extracting device for extracting an object from a video image of a moving image has the following configuration.

(1) A means for acquiring a video, a means for detecting an object area from the acquired video, a means for calculating an existence likelihood map based on the detection result of the object area, and statistical information for calculating statistical information of each pixel Comprising a calculation means and a background difference calculation means for extracting a foreground region as an object by a background difference method based on the statistical information and the background difference threshold, and the statistical information calculation means adds the past statistical information of each pixel to the current statistical information. The statistical information of this time is obtained by reflecting the pixel value at a predetermined update rate, and the update rate determining means for determining the update rate based on the existence likelihood map is further included. The higher the pixel, the lower the update rate.

(2) A threshold value calculation means for calculating the background difference threshold value based on the existence likelihood map is further provided, and for example, a pixel having a higher existence likelihood of an object is made to have a lower background difference threshold value.

(3) The threshold value calculating means is further adapted to dynamically change the background difference threshold value based on the matching ratio between the detection result of the object area and the calculation result of the background difference calculating means.

(4) The statistical information calculating means comprises means for calculating an average value and a standard deviation based on a history of pixel values, the background difference calculating means, the background area is a single Gaussian distribution based on statistical information. I tried to model with.

(5) The means for calculating the existence likelihood map is configured to calculate the present existence likelihood map by weighting the current detection result of the object area with the previous detection results by a predetermined learning rate m.

(6) The means for detecting the object area detects each of the object areas by a plurality of different detection methods, and integrates the detection results into one.

(7) As a post-processing unit for improving the accuracy of the extracted object based on the existence likelihood map, a noise removing unit that regards a foreground region whose average value of the existence likelihood map is below a predetermined noise threshold as a background region is provided. However, the noise threshold is set to be lower as the size of the foreground region is larger.

(8) As a post-processing unit for improving the accuracy of the extracted object based on the existence likelihood map, a defect filling unit that regards a background region whose average value of the existence likelihood map exceeds a predetermined filling threshold as a foreground region is provided. did.

(9) The means for calculating the existence likelihood map calculates the existence likelihood map for each object to be extracted, and the background difference threshold and the update rate are determined for each existence likelihood map.

(10) further comprising update rate review means for reviewing the update rate based on the extraction result of the object and the existence likelihood map, for example, the update rate determined based on the existence likelihood map, the background in the foreground region. Lower than the area.

According to the present invention, the following effects are achieved.

(1) When the current statistical information is obtained by reflecting the current pixel value in the past statistical information of each pixel at a predetermined update rate, the update rate is determined based on the existence likelihood map. , For example, by setting the range of the background difference threshold set based on the existence likelihood map to be lower for pixels having a higher existence likelihood of an object, a stationary object is gradually recognized as a background. It becomes possible to solve the technical problem that it is difficult to be recognized as an object.

(2) Since the background difference threshold is calculated based on the existence likelihood map, the background area is erroneously detected as the foreground area by setting the background difference threshold to be lower for pixels with higher object likelihood. Can be hardened.

(3) Since the background difference threshold is dynamically changed based on the ratio of the area determined to be the foreground and the area determined to be the background, for example, the background difference set based on the existence likelihood map. If the threshold range is set lower for pixels having a higher likelihood of existence of an object, the foreground region can be made less likely to be erroneously detected as the background region.

(4) The average value and the standard deviation are calculated based on the history of pixel values, and the background difference calculation means models the background area with a single Gaussian distribution based on statistical information, so that the background area can be regarded as an object. Can be accurately extracted.

(5) Since the detection result up to the previous time is weighted by the predetermined learning rate m to the detection result of this time in the object area to calculate the existence likelihood map of this time, omission of detection of the object may occur in some frames. Even if it occurs, if the object is detected in the frames before and after that, it is possible to prevent the object from being overlooked.

(6) When detecting the object area from the video, the object detection is executed by each of a plurality of different algorithms, and the detection results are integrated into one. Will be possible.

(7) Since the foreground area in which the average value of the existence likelihood map is lower than a predetermined noise threshold is considered as a background area, a noise removing means is provided, so that the accuracy of object extraction is reduced due to the background area being extracted as the foreground area. Will be able to recover.

(8) The loss of accuracy of object extraction caused by the foreground region being extracted as the background region because the background region in which the average value of the existence likelihood map exceeds the predetermined threshold for filling is considered as the foreground region. Will be able to recover.

(9) If the existence likelihood map is calculated for each object to be extracted and the background difference threshold and the update rate are set for each existence likelihood map, the existence likelihood map becomes unique to each object. The accuracy of can be improved.

(10) Since the means for reviewing the update rate based on the extraction result of the object and the existence likelihood map is further provided, for example, the range of the update rate determined based on the existence likelihood map is set to the background area in the foreground area. If it is set lower than this, it is possible to solve with a higher degree of certainty the technical problem that a stationary object is gradually recognized as a background and is thus difficult to be recognized as an object.

It is a functional block diagram of the object extraction device which concerns on 1st Embodiment of this invention. It is a figure showing a result of object detection by Mask R-CNN method. It is a figure showing the result of object detection by the OpenPose method. It is the figure which compared the result of object detection. It is a figure showing the method of noise removal in a post-processing part. It is a figure showing the method of filling up a deficient hole in a post-processing part. It is the figure which showed the result of missing hole filling in comparison with the prior art. It is a figure showing an example of extraction of an object. It is a functional block diagram of an object extraction device concerning a 2nd embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a configuration of a main part of an object extraction device according to a first embodiment of the present invention, and here, an example of application to an environment in which a plurality of cameras is installed will be described. However, the number of cameras may be only one.

The camera image acquisition unit 1 acquires camera images of moving images from a plurality of cameras cam having different fields of view. The object area detection unit 2 includes a plurality of detection units 21 and 22 having different object detection methods, and uses a plurality of object detection methods typified by deep learning, in which the object area N(x , y) is detected.

In the present embodiment, in order to increase the reliability of object detection, the Mask R-CNN method disclosed in Non-Patent Document 4 is provided as the first detecting unit 21, and the second detecting unit 22 is provided as a non-patent document. The OpenPose method disclosed in No. 6 is provided, each detection method is used together, and the detection results are integrated to improve the reliability of object detection.

Further, in the present embodiment, it is assumed that the object is extracted from the camera image of the volleyball relay, and the objects to be extracted are limited to two types, that is, the player and the ball, and only the player is extracted in the object detection by the OpenPose method. There is.

FIG. 2 is a diagram showing an example of a result of object detection by the Mask R-CNN method, in which silhouette images of a player and a ball are obtained. FIG. 3 is a diagram showing an example of a result of object detection by the OpenPose method, and skeleton information is obtained from the image of each player.

It should be noted that the method of detecting an object to be adopted and the combination thereof are not limited to the above-mentioned methods and the combination thereof, and as disclosed in Non-Patent Document 7, a rectangle including an extraction target object is acquired. Any algorithm may be adopted, or a detection method based on an image feature amount such as HOG (Histograms of Oriented Gradients) feature amount may be employed. When performing object detection based on deep learning, etc., it is necessary to have a trained model that has been trained using the target object in advance, so it is assumed that this model is calculated and prepared in advance. ..

Returning to FIG. 1, the existence likelihood map calculation unit 3 calculates the probability (likelihood) that an object exists at each position of the frame image based on the result of the object area detection, and calculates the existence likelihood on the frame image. Existential likelihood map E(x, y) representing the distribution of degrees is calculated.

In the present embodiment, as shown in the following equations (1) and (2), the existence likelihood map E _t (x, y) at this time (time t) is the existence likelihood map at the previous time (time t−1). It is calculated as a weighted sum of the calculation result of the map E _t-1 (x, y) and the current detection result N(x, y) of the object region based on the predetermined existence likelihood map learning rate m.

Regarding the existence likelihood map learning rate m, the past existence likelihood map value _Et-1( x, y) is propagated to the next frame as a countermeasure when a recognition failure occurs in the object area detection unit 2 or the like. The ratio is shown. N(x, y) represents a value obtained by integrating the detection results obtained from the object area detection unit 2 at time t, R _i represents the detection result obtained from each detection method, and k _i represents the influence of each detection method. 5 shows predetermined parameters for adjusting the ratio. However, when t=0 (first frame), the term of E _t-1 (x, y) in Eq. (1) is calculated as 0.

In this embodiment, since it is using Mask R-CNN method and OpenPose method as the object detecting method, I = 2 becomes, R ₁ (x, y) detected result by the Mask R-CNN method as is, R ₂ ( The detection results of the OpenPose method are respectively substituted as (x, y). In the above formula (2), each detection result is weighted and the sum thereof is calculated, but N(x, y) may be calculated by calculating a product instead of the sum.

In the present embodiment, in the result R ₁ (x, y) of the Mask R-CNN method, the position where it is determined by the Mask R-CNN method that the extraction target object exists is 1 and the position where it does not exist is 0. In addition, in the result R ₂ (x, y) of the OpenPose method, 1 is set for a portion at a constant distance from each extracted skeleton, and 0 is set for other portions. When the object region detection unit 2 extracts the structural information of an object such as a skeleton, even if weighting is performed so that the value of R ₂ (x, y) becomes smaller as the distance from the central portion becomes larger. Good.

Further, when the object area detection unit 2 has a mechanism capable of calculating the existence probability of the object to be extracted when detecting the object, the existence probability is reflected in the value of R _i (x, y). May be.

In addition, regarding the existence likelihood map E(x, y), when there are a plurality of extraction target objects such as players and balls, a existence likelihood map may be created for each object. At that time, there is a possibility that more accurate foreground extraction can be realized by changing the set threshold value or the update rate range for each object. For example, erroneous extraction of the foreground can be reduced by setting the threshold value range of the object that is likely to be easily extracted to be high and the threshold value range of the object that is difficult to be extracted to be low.

The threshold calculation unit 4 refers to the existence likelihood map E(x, y) and separates a threshold T(x, y) when separating each region of the frame image into a foreground and a background based on each pixel value. Is dynamically determined for each pixel. In the present embodiment, a low threshold is set in a region where the existence likelihood of an object is high, and a high threshold is set in a region where the existence likelihood of an object is high. T(x, y) is determined for each pixel. As a result, the region in which the object to be extracted exists can be easily determined to be the foreground.

Here, T _min and T _max are the minimum and maximum values that the separation threshold T(x, y) can take, and may be manually determined in consideration of the target scene, but the object region detection The match ratio between the detection result of the part 2 or the value E _t (x, y) of the existence likelihood map and the area actually extracted as the foreground is calculated, and when the match ratio is low, the threshold range setting is successful. A mechanism may be provided for automatically determining T _min and T _max so that it is determined that the matching ratio has not been improved and the matching ratio is improved.

If the existence likelihood map E(x, y) is created for each extraction target object, T _min and T _max may also be set for each extraction target object (player and ball). Then, the final threshold value of each pixel may be calculated as an average value or maximum value of T(x, y) of each object.

The statistical information calculation unit 6 calculates the statistical information of pixels by statistically reflecting the past pixel value on the current pixel value for each pixel of the acquired frame image. In this embodiment, for each pixel, the average value calculation unit 61 calculates the average value u(x, y) of pixel values based on the following formula (4), and the standard deviation calculation unit 62 calculates based on the following formula (5). Then, the standard deviation σ(x, y) of the pixel values is calculated.

Since the statistical information calculation unit 6 repeats the calculation of statistical information at each time t when a new frame image is acquired, each statistical information u(x, y), σ(x, y) is updated in frame units. It will be. U(x, y) is the ratio (update rate) of reflecting the past statistical information on the current pixel value when the statistical information is calculated, and the update rate determining unit 5 sets the existence likelihood map E(x , y) is used as a parameter, and calculation is performed based on the following equation (7).

U _min and U _max are respectively the minimum value and the maximum value that the update rate can take, and in the present embodiment, a low update rate is set in a region where the likelihood of existence of an object is high. Therefore, even if the object to be extracted remains stationary for a long time, it is possible to prevent the object to be extracted from being chipped if the value of the existence likelihood map is kept high.

It should be noted that U _min and U _max may be designed to have different update rates according to the foreground/background discrimination result of each pixel obtained by the background difference calculation unit 7 described later. Generally, it is desirable to set a high update rate for pixels determined to be the background and a low update rate for pixels determined to be the foreground.

The background difference calculation unit 7 calculates the statistical information (in this embodiment, μ _t (x, y) and standard deviation σ _t (x, y)) of each pixel for each frame image, the separation threshold T(x, y), and Foreground/background discrimination is performed in pixel units based on the pixel value I _t (x, y), and the discrimination result is output in, for example, a mask format.

In this embodiment, as in Non-Patent Document 2, consider modeling a background with a single Gaussian model. Regarding the color space, the processing is described in the same YUV color space as in Non-Patent Document 2, but the same processing can be performed even if another color space such as RGB is targeted as the color space. However, if the color space obtained by the camera image acquisition unit 1 and the color space to be calculated by the background difference calculation unit 7 are different, it is assumed that the input image has a mechanism for converting the color space. To do. Then, the pixel (x, y) that satisfies the following expression (8) is determined to be the background.

Here, z is a parameter for adjusting how many times the standard deviation is determined as the background, and T(x, y) is a threshold value calculated by the threshold value calculation unit 4. Therefore, the larger T(x, y) is, the more likely it is to be judged as the background. Further, in the present embodiment, it is stated that the calculation is performed in the YUV color space. However, in the case of having a plurality of color spaces, the calculation of the above conditional expression is performed independently for each color space, and all color spaces are calculated. If the conditions are satisfied, the pixel is determined to be the background.

Furthermore, in the above equation (8), the standard deviation term and the threshold value term are separated, but in practice, the constant value z of the standard deviation term is changed according to T(x, y) and the value of the existence likelihood map. May be provided with a function for adjusting.

The post-processing unit 8 includes a noise removing unit 81 that removes noise based on the existence likelihood map E _t (x, y) and a defect filling unit 82 that fills in a defective region, and the background difference calculating unit 7 With respect to the mask output by, the noise removal by a filtering process such as a median filter or the process of removing fine noise by repeating the dilation and erosion of the contour is performed.

Regarding the noise removing unit 81, in the background subtraction method of Non-Patent Document 2, for the mask calculated using the background difference, the combined foreground region is regarded as one block, and the size of the outline frame for each block and Noise is removed by checking the aspect ratio.

However, in such a conventional method, when a small object such as a ball enters, for example, a value smaller than the ball has to be set as a noise removal parameter in order to prevent the ball from disappearing, and it is effective. Is difficult.

In addition, when the player's mask is divided for some reason, it may be deleted if the size of the divided portion is small. Therefore, in the present embodiment, not only the size of the divided chunk but also the value of the existence likelihood map E _t (x, y) is used to remove noise.

For example, as shown in FIG. 5, three blocks P _j (P ₁ , P ₂ , P ₃ : j are block identifiers) are added to the primary mask [(a) in the figure] output from the background difference calculation unit 7. , The average value d _u of the existence likelihood map inside the blocks P ₁ , P ₂ , P ₃ [(b) in the same figure] is calculated. Then, by removing only the mass P ₁ whose average value _du is lower than the noise removal threshold d _ref (for example, d _ref =0.5), the other masses P ₂ and P ₃ are left, It is possible to leave a small area at a position with a high likelihood [(c) in the same figure], and it is possible to perform highly accurate noise removal.

The noise removal threshold value d _ref may be a fixed value, or may be reduced as the size of the target region increases, and even if a mechanism is provided that can perform noise removal only when it can be reliably determined that the object is not the extraction target object. Good.

As shown in FIG. 6, a small area (a missing area) in which the foreground is erroneously determined to be the background may be formed in the missing hole filling portion 82 so as to be surrounded by the foreground area (white portion). Such a defective area is generated by erroneously determining a part of the object as the background, for example, when the color of the clothes of the person has a portion having the same color as the background. In the present embodiment, the average value of the existence likelihood map E(x, y) inside the loss area is calculated in the same manner as the noise removing unit 81, and if the average value exceeds a predetermined threshold value, padding is performed. Repair the defective area.

FIG. 7 shows a case in which the defect region is filled up by increasing and decreasing the threshold value according to the method of Non-Patent Document [(a) and (b) in the figure], and the existence filling map E(x, FIG. 8 is a diagram comparing with the case [filled in (c) in the figure] when filling is performed based on y).

When the threshold value is set to a low value by the method of Non-Patent Document 2, the number of missing objects (players) is small, but the background of the floor or signboard is erroneously determined to be the foreground. Further, when the threshold value is set to a high value by the method of Non-Patent Document 2, it becomes apparent that the lack of objects (players) is scattered, and there is a limit to the prevention of the loss in the setting of the threshold value.

On the other hand, in the present embodiment, since the post-processing unit 8 performs noise removal and defect filling based on the existence likelihood map E(x, y), the background area is reliably removed and the object is cleaned. You can see that it has been extracted.

The output unit 9 outputs the resulting video (image) based on the information of the background area calculated by the background difference calculation unit 7 or the post-processing unit 8. The image output here may be a color image obtained by masking the input image as shown in FIG. 7, or may be a binary image for determining the background/foreground as shown in FIG. It may be a binary mask image.

According to the present embodiment, the range of the update rate of the background statistical information set based on the existence likelihood map becomes lower as the pixel having the higher existence likelihood of the object becomes lower, so that the stationary object gradually becomes the background. It becomes possible to solve the technical problem that it is difficult to recognize as an object because it is recognized.

FIG. 9 is a functional block diagram showing a configuration of a main part of an object extracting device according to the second embodiment of the present invention, and the same reference numerals as those used above represent the same or equivalent parts, and therefore the description thereof will be omitted. To do.

In the present embodiment, the update rate determining unit 5 includes an update rate reviewing unit 51, and the update is performed based on a comparison result between the output of the background difference calculating unit 7 and the existence likelihood map E(x, y). The feature is that the rate U(x, y) is reviewed. The output mask used by the update rate review unit 51 to review the update rate is not limited to the output of the background difference calculation unit 7, and a mask output by the post-processing unit 8 may be used.

The update rate reviewing unit 51 sets a higher update rate for the pixels determined to be the background by the background difference calculation unit 7 and a lower update rate for the pixels determined to be the foreground. Then, the update rate U _fore (x, y) of the pixel determined to be the foreground by the background difference calculation unit 7 is calculated based on the following equation (9).

On the other hand, the update rate U _back (x, y) of the pixel determined to be the background by the background difference calculation unit 7 is calculated based on the following expression (10).

Here, U _minfore and U _minback are the minimum values of the update rates that can be taken by the pixel determined to be the foreground and the pixel determined to be the background, respectively, and U _minfore <U _minback . U _maxfore and U _maxback are the maximum values of the update rates that can be taken by the pixel determined to be the foreground and the pixel determined to be the background, respectively, and U _maxfore <U _maxback .

According to the present embodiment, the range of the update rate determined based on the existence likelihood map is set to be lower in the foreground area than in the background area, so that the stationary object is gradually recognized as the background. It becomes possible to solve the technical problem of being difficult to be recognized as an object with higher accuracy.

1... Camera image acquisition unit, 2... Object region detection unit, 3... Existence likelihood map calculation unit, 4... Threshold value calculation unit, 5... Update rate determination unit, 6... Statistical information calculation unit, 7... Background difference calculation unit, 8... Post-processing section, 9... Output section, 21... First detection section, 22... Second detection section, 81... Noise removal section, 82... Defect filling section

Claims (18)

In an object extraction device that extracts objects from video images,
Means to capture video,
Means for detecting the object area from the acquired video,
Means for calculating an existence likelihood map of the object based on the detection result of the object area,
Statistical information calculating means for calculating statistical information of each pixel,
A background difference calculation means for extracting the foreground region as an object by the background difference method based on the statistical information and the background difference threshold value,
The statistical information calculating means obtains the current statistical information by reflecting the current pixel value in the past statistical information of each pixel at a predetermined update rate,
The object extracting apparatus further comprising an update rate determining means for determining the update rate based on the existence likelihood map. The object extraction device according to claim 1, wherein the update rate determination unit lowers the update rate for pixels having a higher likelihood of existence of an object. 3. The object extracting apparatus according to claim 1, further comprising a threshold value calculating means for calculating the background difference threshold value based on the existence likelihood map. The object extraction device according to claim 3, wherein the threshold value calculation unit lowers the background difference threshold value for a pixel having a higher likelihood of existence of an object. 5. The object according to claim 3, wherein the threshold value calculation means dynamically changes the background difference threshold value based on a matching ratio between the detection result of the object area and the calculation result of the background difference calculation means. Extractor. The means for calculating the existence likelihood map calculates the present existence likelihood map by weighting the current detection result of the object region with the previous detection results by a predetermined learning rate. 6. The object extraction device according to any one of 1 to 5. 7. The object extracting apparatus according to claim 1, wherein the means for detecting the object area detects each of the object areas by a plurality of different detection methods and integrates the respective detection results into one. .. 8. The object extracting apparatus according to claim 1, further comprising post-processing means for improving the accuracy of the extracted object based on an existence likelihood map. 9. The object extracting apparatus according to claim 8, wherein the post-processing unit includes a noise removing unit that regards a foreground region whose average value of the existence likelihood map is below a predetermined noise threshold as a background region. The object extraction device according to claim 9, wherein the noise threshold is set to be lower as the size of the foreground region is larger. 11. The object according to claim 8, wherein the post-processing unit includes a defect filling unit that regards a background region in which the average value of the existence likelihood maps exceeds a predetermined filling threshold as a foreground region. Extractor. 12. The object extraction apparatus according to claim 1, wherein the means for calculating the existence likelihood map calculates an existence likelihood map for each object to be extracted. The object extraction device according to claim 12, wherein the background difference threshold and the update rate are determined for each existence likelihood map. 14. The object extracting apparatus according to claim 1, further comprising update rate reviewing means for reviewing the update rate based on the extraction result of the object and the existence likelihood map. 15. The object extracting apparatus according to claim 14, wherein the update rate reviewing unit sets the update rate determined based on the existence likelihood map to be lower in the foreground area than in the background area. 16. The object extracting apparatus according to claim 1, wherein the statistical information calculating means includes means for calculating an average value and a standard deviation based on a history of pixel values. The object extraction device according to any one of claims 1 to 16, wherein the background difference calculation means models the background area with a single Gaussian distribution based on the statistical information. In the object extraction method for extracting objects from video images,
The procedure of acquiring the image and detecting the object area,
A procedure for calculating the existence likelihood map of the object based on the detection result of the object area,
A procedure for calculating the statistical information of each pixel,
A step of extracting a foreground region as an object by the background difference method based on the statistical information and the background difference threshold, and in the procedure of calculating the statistical information of the pixel, the current pixel value is set in the past statistical information of each pixel. The object extraction method is characterized in that the statistical information of this time is obtained by reflecting the update rate with the update rate, and the update rate is determined based on the existence likelihood map.