JP7121708B2 - Object extractor, method and program - Google Patents

Description

The present invention relates to an object extraction device, method, and program, and more particularly to an object extraction device, method, and program for extracting an object from each frame of a video by background difference calculation using a background model obtained by statistically processing past frames as a background. Regarding the program.

Many techniques for separating a foreground region and a background region have been proposed mainly for the purpose of detecting a moving object in an image and producing a free-viewpoint video as shown in Non-Patent Document 1. Among them, the approach of modeling the features of the background based on a background image with no foreground taken in advance and extracting the area where the difference between the background model and the input image is large as the foreground is called the background subtraction method.

As an example of the background subtraction method, Non-Patent Document 2 discloses a method of modeling the background using a mixed Gaussian distribution obtained by mixing a plurality of Gaussian distributions. Non-Patent Document 2 realizes separation of the foreground and the background by extracting pixels having a certain difference or more as the foreground from the background modeled by the mixed Gaussian distribution.

The background model used in Non-Patent Document 2 has a mechanism for gradually updating each frame. As a result, even if the background portion changes, the background model can be maintained according to the situation by gradually learning the characteristics of the background.

In Non-Patent Document 3, the accuracy of the background subtraction is improved by adaptively changing the threshold for foreground extraction that is set when performing the background subtraction for each pixel according to the human position obtained by tracking the person. A method for increasing is disclosed. In Non-Patent Document 3, it is expected that the accuracy of the technology using background subtraction disclosed in Non-Patent Document 2 can be further improved by changing parameters based on such person recognition and tracking information.

Patent Document 1 relates to an invention that realizes highly accurate foreground extraction by dynamically changing both the threshold value and the update rate for each pixel when performing the background subtraction method based on the result of object recognition. is.

In Japanese Patent Laid-Open No. 2004-100001, the background difference threshold value of a region in which an object to be extracted as the foreground is likely to exist is lowered based on the result of object recognition, thereby making it easier to extract the object to be extracted as the foreground. In addition, by lowering the update rate of the background model for the pixels where the object exists based on the result of object recognition, it is possible to prevent the static object from being chipped.

Patent application No. 2019-009705

H. Sankoh, S. Naito, K. Nonaka, H. Sabirin, J. Chen, "Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes", Proceedings of the 26th ACM international conference on Multimedia, pp. 1724-1732 (2018) C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252 Vol. 2 (1999). Kenji Terabayashi, Kazunobu Umeda, Alessandro Moro, "Real-time Adaptive Threshold Processing of Background Subtraction Using Human Tracking Information", IEEJ General Industry Research Group, GID-09-17, pp.89-90(2009). Gunnar Farneback, "Two-frame motion estimation based on polynomial expansion." In Proceedings of the 13th Scandinavian conference on Image analysis (SCIA'03), pp.363-370, 2003. J. F. Henriques, R. Caseiro, P. Martins and J. Batista, "High-Speed Tracking with Kernelized Correlation Filters," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 3, pp. 583-596, 2015. K. He, G. Gkioxari, P. Dollar and R. Girshick, "Mask R-CNN," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980-2988 (2017). J. Redmon and A. Farhadi,"YOLO9000: Better, Faster, Stronger," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6517-6525 (2017).

The inventors of the present invention used silhouettes extracted by a background subtraction-based method as disclosed in Non-Patent Documents 2 and 3 and Patent Document 1, and used a free-viewpoint silhouette represented by Non-Patent Document 1. I made a video and made a study.

As a result, in the production process of the free-viewpoint video technology disclosed in Non-Patent Document 1, silhouette images are created using the background subtraction method, and then the intersection in the three-dimensional space is calculated based on each silhouette image. By doing so, a visual volume is generated and a 3D model of the target object is created. It was confirmed that the accuracy of silhouette extraction at this time greatly affects the quality of the free-viewpoint video.

The method disclosed in Non-Patent Document 2 has the strength of being able to robustly extract silhouettes with respect to gradual changes in the background and regular changes in the background part by statistically modeling the background and gradually updating it. was there. However, it has been difficult to apply it to a scene in which the background is complicated and changes abruptly.

An example of a scene with a complex background is a scene in which a liquid crystal display for displaying advertisements that frequently switch is placed behind the player when you want to extract players and balls in a sports game, or a scene in which bad weather is used. Scenes in which the background situation constantly changes greatly can be cited.

In such a scene, the background changes abruptly, and the change of the background often has no regularity. Therefore, there is a high possibility that the background is mistakenly extracted as the foreground. The method of Non-Patent Document 2 is insufficient in accuracy to perform extraction in these scenes that are difficult to extract.

In order to solve such technical problems, as in Non-Patent Document 3 and Patent Document 1, there is a method of performing person tracking and person recognition, and changing the threshold value and update rate of the background subtraction method based on the results. had been proposed. However, Non-Patent Literature 3 only discloses a mechanism for reducing the threshold value inside a rectangle obtained by tracking to facilitate extraction. Therefore, although it becomes easier to extract the target object, a new problem may arise in that a large amount of noise is likely to occur in the area where the threshold is lowered.

On the other hand, in Patent Literature 1, based on the result of person recognition, not only the threshold value but also the update rate of the background model are changed to perform highly accurate extraction. This adjustment of the threshold value and the update rate is performed only based on the result of person recognition for each frame, and the threshold value is set low especially in areas where persons are continuously recognized over a plurality of frames.

However, in situations where a person who has been standing still for a long time suddenly starts to move, the threshold for pixel positions where the object has been standing still for a long time continues to be set low for a while, so noise tends to appear in this area. In addition, there is a problem that chipping is likely to occur when an object moves to an area where the object has not been recognized, ie, an area with a high threshold value. As described above, Patent Document 1 has a problem that noise and chipping are likely to occur with respect to irregular movement of the target object.

It is an object of the present invention to solve the above technical problems and to provide an object extraction device, method, and program that enable robust object extraction even when the object moves violently, such as when the object moves irregularly or stands still. It is in.

In order to achieve the above object, the present invention provides an object extracting apparatus for extracting an object from each frame of a video by background difference calculation using a background model obtained by statistically processing past frames as a background, and having the following configuration. It is characterized by the fact that it is equipped.

(1) means (102) for detecting an object region from each frame; means (103) for acquiring displacement information of the object region between frames; means (104a) for moving to a corresponding pixel region of the current frame to determine a background difference threshold to be used for the current frame; and background difference calculation for comparing the difference between the background model and the current frame with the updated background difference threshold. Means (105) for identifying each pixel of the current frame as either background or foreground, and means (106) for updating the background model based on the current frame.

(2) update rate determination means (104b) for determining the update rate based on the displacement information when the means (106) for updating the background model updates the background model by reflecting the current frame at a predetermined update rate; was further provided.

(3) Based on the displacement information, the update rate determination means (104b) adjusts the update rate to a higher value for a pixel with a higher displacement speed.

(4) The update rate determining means (104b) adjusts the update rate of the pixel where the object is detected to be low based on the object detection result for the current frame.

(5) means (106) for updating the background model updates the pixels identified as the foreground at a lower update rate than the pixels identified as the background, based on the identification result of the identifying means (105); I made it

(6) The means for determining the background difference threshold adjusts the background difference threshold determined based on the displacement information to be lower for the pixels where the object is detected based on the result of object detection for the current frame.

(7) It further comprises means for initializing the update rate of each pixel of the movement source of which the background difference threshold has been moved based on the displacement information.

(8) Further provided is means for initializing the background difference threshold of each pixel of the movement source of which the background difference threshold has been moved based on the displacement information.

(1) Based on the displacement information of the object area between frames, the background difference threshold used in the previous frame is moved to the corresponding pixel area in the current frame to determine the background difference threshold used in the current frame. Robust object extraction becomes possible even when the movement is intense, such as moving or standing still irregularly.

(2) The means for updating the background model determines the update rate when updating the background model by reflecting the current frame at a predetermined update rate based on the displacement information. can be adaptively optimized.

(3) The update rate determining means adjusts the update rate to a higher value as the displacement speed of a pixel increases. is assigned a high update rate, which makes it easier to sharpen contours.

(4) Since the update rate determining means adjusts the update rate of the pixel where the object is detected to be low, it is possible to prevent the identification result of the stationary object from being deficient.

(5) The means for updating the background model updates the pixels identified as the foreground at a lower update rate than the pixels identified as the background, so that it is possible to prevent the identification results of stationary objects from being deficient. become.

(6) The means for determining the background difference threshold adjusts the background difference threshold determined based on the displacement information to be lower for the pixels where the object is detected, so that the detected area of the object is identified as the foreground by the background difference calculation. easier to do.

(7) Since means for initializing the update rate of each pixel of the movement source whose background difference threshold has been moved based on the displacement information is provided, noise occurs due to the update rate being maintained at a low value. can be prevented.

(8) The background difference threshold is maintained at a low value because a means for initializing the background difference threshold of each pixel whose background difference threshold has been moved based on the displacement information is provided. It becomes possible to prevent the occurrence of noise.

1 is a functional block diagram showing the configuration of main parts of an object extraction device according to an embodiment of the present invention; FIG. It is a figure showing an example of object detection. FIG. 4 is a diagram showing an example of object tracking; FIG. 10 is a diagram showing a method of determining a background difference threshold; FIG. 10 is a diagram showing an example in which noise occurs due to the background difference threshold remaining in the pixel from which the background difference threshold is moved. FIG. 10 is a diagram showing an example of initializing the background difference threshold of the movement source after moving the background difference threshold; FIG. 10 is a diagram showing a display example of a background/foreground identification result based on background difference calculation; It is a flow chart showing the operation of one embodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of main parts of an object extraction device 1 according to one embodiment of the present invention. In the present embodiment, a background model obtained by statistically processing past frame images is used as the background when an object is extracted from each frame of video by background difference.

Such an object extraction device 1 can be configured by installing an application (program) that realizes each function described later on a general-purpose computer or server equipped with a CPU, memory, interfaces, and a bus connecting them. Alternatively, a part of the application can be configured as a dedicated machine or a single-function machine that is hardware or programmed.

The frame image acquisition unit 101 acquires the frame image I from the video captured by the camera 2 and provides it to the object detection unit 102 , the background difference calculation unit 105 and the background model update unit 106 . This frame image I may be an image captured by a camera or the like and stored in a hard disk or the like, and the stored image may be used as an input for processing. The object detection unit 102 detects an object from each frame, and outputs the detection result O(x, y) in the current frame I _N as an object distribution L _N (x, y), as shown in an example in FIG. . Here, N indicates the frame number, and (x, y) indicates the position of each pixel within the frame. In the object distribution L _N (x, y), a high value is registered for pixels where an object is detected, and a low value is registered for pixels where no object is detected, in the range of 0 to 1.

In this embodiment, an image feature amount is extracted from each frame of a camera video of an object to be detected under various environments, and each object is detected using a learning model that normalizes the results of deep learning.

For example, if the camera video is a video of a ball game, the ball and the player are detected as objects. For such object detection, as disclosed in Non-Patent Document 6, a deep learning-based extraction method that can extract even the shape of the object, as disclosed in Non-Patent Document 7, a rectangular region of the object is used. can be applied. Alternatively, each object may be extracted based on an image feature amount such as HOG (Histograms of Oriented Gradients) feature amount.

Note that if the object detection unit 102 is configured to be able to identify the class of each object, the information on the identification class may also be included in the output. Identification class information is information for identifying whether an object is, for example, a "person" or a "ball".

Further, if the object detection unit 102 is configured to be able to calculate the likelihood indicating the likelihood of the detection result, this likelihood is reflected in the object detection result O (x, y), and background difference parameter determination (described later) is performed. It may be used for parameter determination in section 104 .

Furthermore, the object distribution L _N (x, y) is the object distribution L _{N -1} of the previous frame I _N -1 and the object detection result O (x, _y ) may be obtained as a weighted sum obtained by adding them at a predetermined ratio m. As a result, even if an object recognition error occurs in one frame, the effect of the error on other frames can be reduced.

The displacement information acquisition unit 103 compares the object detection result for the current frame I _N with the object detection result for the previous frame I _N-1 , and acquires the displacement direction and displacement amount of each object between frames as displacement information. do. At this time, if the displacement information is obtained by the calculation of the optical flow disclosed in Non _- Patent Document ₄ , the prior detection of the object is generally unnecessary. can be obtained. Although displacement information can be calculated relatively quickly and easily using optical flow, the accuracy is not sufficient.

Another approach is to obtain displacement information by tracking the detected object, as shown in FIG. In this case, starting from the object area detected by the object detection unit 102, the position of the object is tracked between frames using the object tracking technique disclosed in Non-Patent Document 5. At this time, an approach may be adopted in which object detection is also performed in parallel for each frame, and if the object detection result overlaps with the tracking result, the detection result is used as the start of new tracking, and the tracking is repeatedly performed. Based on these approaches, the displacement information acquisition unit 103 can acquire the displacement direction and displacement amount of each pixel between frames.

The background difference parameter determination unit 104 includes a background difference threshold determination unit 104a, an update rate determination unit 104b, and an initialization unit _104c . Determine the background difference parameters, such as the update rate of the background model).

The background difference threshold determining unit 104a, as shown in FIG. 4, sets the background difference threshold applied to the object area of the previous frame I _N-1 to each of the corresponding areas of the current frame I _N based on the displacement information. Moving to the pixel determines the background difference threshold to apply to the current frame I _N . For example, the displacement information from the previous frame I _N-1 to the current frame I _N is expressed as a displacement vector (Δx, Δy), and the background difference threshold applied to the previous frame I _N-1 is expressed as T _N-1 (x, y). Then, the displacement-based threshold T _{N_MOV} (x, y) applied to the current frame I _N is obtained by the following equation (2).

Note that in the case where the pixels of two objects that were independent in the previous frame I _N-1 move to the same pixel in the current frame I _N , it is possible that the threshold values will move to the same pixel from two locations. Conceivable. In such cases, measures such as preferentially choosing the mean or the smaller value of each threshold are taken to obtain the displacement-based threshold T _{N_MOV} (x, y).

In addition, if the front-back relationship of a plurality of overlapping objects can be determined based on depth estimation or the like, a mechanism may be provided that preferentially reflects the threshold value of the front moving object. At this time, for pixels that do not correspond to the movement destination from the previous frame I _N -1, the threshold T _{N-1_} (x, y) of the corresponding pixel in the previous frame I _N-1 is calculated based on the following equation (3). can be adopted as is.

The background difference threshold determination unit 104a further calculates the background difference threshold T _{N_MOV} (x, y) based on the movement, and then converts the object detection result L _N (x, y) output from the object detection unit 102 to T _{N_MOV} An adjustment value T _{C (} x, y) to be reflected in (x, y) is calculated by the following equation (4). T _max is a constant that indicates the maximum amount of change in the threshold between frames, and is set in advance.

In other words, it is desirable to lower the threshold in order to make it easier to determine that the pixels in which the object exists according to the object distribution L _N (x, y) are in the foreground. Therefore, in the present embodiment, a negative adjustment value T _C (x, y) is obtained for a pixel where an object exists, and a positive adjustment value T _C (x, y) is obtained for a pixel where an object does not exist. there is

The background difference threshold determination unit 104a applies the displacement-based threshold T _{N_MOV} (x, y) and the adjustment value T _C (x, y) to the following equation (5) to determine the final value to be applied to the current frame I _N . A realistic background difference threshold T _N( x, y) is obtained.

As described above, in the present embodiment, a low threshold value is set for an area where there is a high probability that an object exists, and a high threshold value is set for an area where there is a low possibility that an object exists. to be born.

The update rate determining unit 104b adds the pixel information of the current frame I _N to the background model modeled by the background model updating unit 106, which will be described later, by statistically processing each frame up to the previous frame (to I _N-1 ). A rate of reflection (update rate U _N (x, y)) when updating the background model up to the current frame (˜I _N ) is determined for each pixel based on the displacement information.

In this embodiment, in order to prevent the update rate U _N ( _x , y) from changing significantly between frames, the update rate U Based on _N −1(x, y) and the correction value U _C (x, y), the update rate U _N (x, y) for updating the background model used in the background subtraction calculation for the current frame I _N is Calculated by the following formula (6).

However, since the update rate U _N (x, y) has a value range of 0 to 1, the value is corrected to 0 if it is less than 0, and 1 if it is greater than 1. . The correction value U _C (x, y) is the amount of change in the update rate of each pixel in the current frame I _N and is obtained by the following equation (7).

Here, U _max is a constant that indicates the maximum amount of change in the update rate between frames, and is set manually in advance. M _N (x, y) is the displacement amount of each pixel between frames, that is, the index value of the displacement speed. Take the normalized value. Thus, U _C (x, y) is U _max when M _N (x, y) is 1, and U _C (x, y) is -U _max when M _N (x, y) is 0. As a result, the update rate changes at most by U _max according to M _N (x, y).

As described above, in this embodiment, a large update rate is assigned to a pixel with a high displacement speed, and a small update rate is assigned to a pixel with a low displacement speed. etc. will be less likely to occur. Also, since a high update rate is assigned to a fast-moving object, it becomes easier to sharpen the contour.

Also, in cases where another object overlaps the same position during tracking, there may be cases where two M _N (x, y) are obtained, but in this case, the average or maximum amount M _N ( x, y) can be adopted. Also, similar to the determination of the threshold value, if the anteroposterior relationship of multiple overlapping objects can be determined based on depth estimation, etc., M _N (x, y) of the moving object in front is preferentially reflected. A mechanism may be provided to allow

Furthermore, similar to the threshold, the object distribution L _N (x, y) may be reflected in the above equation (7), and control may be added to lower the update rate when an object exists. By lowering the update rate of pixels in which an object exists, it is possible to expect the effect of preventing missing of a stationary object.

If the object detection unit 102 can identify the class of each object, the maximum values U _max and T _max may be set for each identification class. As a result, for example, in a situation where the skin color of a person is very similar to the color of the floor and is easily chipped, and the ball is a color that is far from the background and is difficult to chip, T _max (T _{max_person} ) of the person class can be By designing T _max (T _{max_ball} ) of the ball class to be large and small, the amount of change in the threshold becomes larger in cases where the person class is recognized, and the area where it is thought that a person is likely to exist. In this case, effects such as facilitating extraction can be expected.

Furthermore, in determining the update rate U _N (x, y) and the threshold T _{N (} x, y), the acquisition of the displacement vector by the rectangle-based tracking method and the pixel-based displacement vector obtained from optical flow, etc. A combined approach may also be used.

For example, when performing rectangle-based tracking, we acquire displacement vectors using optical flow only for the pixels within the rectangle, and set different thresholds and update rates for each pixel within the rectangle. can be obtained. Rectangle-based tracking can only yield one displacement vector per rectangle, but by computing and using pixel-based displacement vectors inside the rectangle, we can adaptively vary the threshold and update rate for each pixel. can be done. This is the weighted sum of the rectangular M _N1 (x, y) and the pixel-based M _N2 (x, y), which is used, for example, in equation (7) _above . This is achieved by calculating

Adding such processing increases the possibility of setting appropriate parameters when the rectangle itself is stationary but the object is moving inside the rectangle. In addition, since the pixel-based displacement vector is obtained only within the rectangle, the calculation can be performed at a higher speed than when the pixel-based displacement vector is obtained for the entire image.

Furthermore, in the object tracking area, the background/foreground discrimination results are more likely to change in the area near the outside of the object tracking area. Processing such as adjustment may be added.

When the threshold T _N-1 (x, y) is moved based on the displacement vector, the initialization unit 104c initializes the threshold and update rate of each pixel (x-Δx, y-Δy) of the movement source. do. This is done in order to prevent noise from easily occurring in the area of the movement source when the object moves, as shown in an example in FIG.

That is, if the threshold and the update rate of the movement source are left as they are, the threshold is kept high and the update rate low for a while, so such noise may occur. Therefore, as shown in FIG. 6, in this embodiment, by initializing the threshold value and the update rate of the movement source, an effect of suppressing the occurrence of noise can be expected.

The background difference calculation unit 105 uses a background model in which each pixel is modeled by a single Gaussian model as the background, and divides each pixel of the current frame I _N acquired by the frame image acquisition unit 101 into the foreground and background by the background difference. Identify one.

In the present embodiment, the background model updating unit 106, which will be described later, sets the average μ _N-1 (x, y) and the standard deviation σ _N-1 on the time axis for each pixel as the background model up to the previous frame I _N -1 . (x, y) is obtained, and pixels (x, y) satisfying the following equation (8) are identified as the background in background subtraction using a Gaussian distribution.

Here, I _N (x, y) is the luminance value of each pixel in the current frame I _N , z is a parameter for adjusting the standard deviation up to which the background is judged to be the background, and the background difference threshold T _N (x , y) is more likely to be judged as background.

The color space of the image used to determine the background difference may be grayscale, RGB, YUV, or other color space. It is desirable to ensure that pixels whose color satisfies the background condition are identified as background.

In addition, although the standard deviation term and the threshold term are separated in the above equation (8), a function to adjust the constant value z of the standard deviation term according to the background difference threshold T _N (x, y) is added. may be provided.

In addition, some post-processing may be performed on the mask output by the background difference calculation unit 105 as a calculation result. Post-processing is a general term for processing for refining the obtained mask by applying filtering or the like to the mask, and is incorporated in many background subtraction algorithms after background subtraction calculation.

Representative examples of post-processing include noise removal by filtering such as a median filter, and processing to remove fine noise by repeating contour dilation and erosion. A foreground area smaller than the size specified by the user may be subjected to noise removal processing such that it is unconditionally treated as the background. If the size of the background area surrounded by the foreground is small, it is filled as the foreground area. etc. may be performed.

The background model updating unit 106 updates the background model used as the background for the next frame I _N+1 based on the calculation result of the background difference for the current frame. In this embodiment, the average value μ _N-1 (x, y) and the standard deviation σ _N-1 (x, y) of each pixel up to the previous frame I _N-1 are combined with the pixel information of the current frame I _N and the It is updated based on the update rate U _N (x, y) determined by the update rate determination unit 104b. That is, the average μ _N (x, y) of each pixel reflecting the luminance value I _N (x, y) of each pixel of the current frame I _N is calculated by the following equation (9).

Furthermore, the standard deviation σ _N (x, y) of each pixel reflecting the luminance value I _N (x, y) of each pixel in the current frame is calculated by the following equations (10) and (11).

Regarding the update rate U _N (x, y), pixels determined to be the foreground are updated at a lower update rate than pixels determined to be the background based on the calculation result of the foreground/background by the background difference calculation unit 105. You can let it be. In general, by setting a high update rate for background pixels and a low update rate for foreground pixels, it is expected that an area determined as the foreground will be prevented from being updated incompletely.

As for the background model, before actually trying to extract the silhouette image, the background model is recorded for a certain period of time in advance, and the background model is used to perform calculations in the background difference calculation unit. may have

The calculation result output unit 107 outputs the background/foreground calculation result by the background difference calculation unit 105 as an image. The output format may be, for example, a color image format obtained by masking each frame with a background pixel area as shown in FIG. 7, or a binary mask image format that allows discrimination between the background and the foreground.

FIG. 8 is a flowchart showing the operation of one embodiment of the present invention. In step S1, the current frame image (current frame I _N ) is acquired from the camera video by the frame image acquisition unit 101 . In step S2, the object detection unit 102 detects an object area from the current frame _IN . In step S3, the displacement information acquisition unit 103 obtains displacement information (Δx, Δy) of the object region between frames based on the object region detected from the previous frame I _N-1 and the object region detected from the current frame I _N . is obtained.

In step S4, as detailed with reference to FIG. 4, _{a displacement- based} A threshold T _{N_MOV} (x, y) is calculated. In step S5, an adjustment value T _C (x, y) is calculated based on the object detection results for the current frame I _N . In step S6, by applying the displacement base threshold T _{N_MOV} (x, y) and the adjustment value T _C (x, y) to the above equation (5), the background difference threshold used in the background difference calculation for the current frame I _N is _TN( x, y) is determined.

In step S7, as described in detail with reference to FIG. 6, the background difference threshold T _N-1 (x, y) applied to the previous frame I _N-1 is moved based on the displacement information. are initialized by the initialization unit 104c. In step S8, the update rate correction value U _C (x, y) is calculated based on the above equation (7). In step _S9 , the _current frame _{I N} The update rate U _N (x, y) for updating the background model used in the background difference calculation for the next frame I _N+1 is determined.

In step S10, a background model (mean μN-1(x, y) and standard deviation σN _- 1( x, y)) is obtained. In step S11, the background difference calculation unit 105 performs background difference calculation for comparing the difference between the pixel information of the current frame I _N and the obtained background model with the background difference threshold T _{N (} x, y). identifies each pixel of the current frame I _N as either background or foreground.

In step S12, the update rate U _N (x, y) determined in step S9 is set such that the pixels determined to be the foreground are higher than the pixels determined to be the background based on the result of the background difference calculation for the current frame I _N . Corrected to update at a lower update rate. In step S13, the background model (mean value μN-1(x, y), standard deviation σN-1(x, y)) used in the background subtraction calculation for the previous frame I _N-1 and the pixel information of the current frame I _N By applying I _N (x, y) and the update rate U _N (x, y) to the above equations (9) and (10), the background model is updated for the next frame I _N+1 .

In step S14, it is determined whether or not the video has ended. If not completed, the process returns to step S1, and the above processes are repeated for the next frame _IN+1 .

1... Object extraction device, 2... Camera, 101... Frame image acquisition unit, 102... Object detection unit, 103... Displacement information acquisition unit, 104... Background difference parameter determination unit, 104a... Background difference threshold determination unit 104b... Update rate determination unit 104c... Initialization unit 105... Background difference calculation unit 106... Background model update unit 107... Calculation Result output part

Claims (15)

An object extraction device for extracting an object from each frame of a video by background difference calculation using a background model obtained by statistically processing past frames as a background,
means for detecting object regions from each frame;
means for obtaining displacement information of an object area between frames;
means for determining a background difference threshold used for the current frame by moving the background difference threshold used for the previous frame to a corresponding pixel region of the current frame based on the displacement information;
means for identifying each pixel of the current frame as either background or foreground by a background difference calculation comparing the difference between the background model and the current frame to the updated background difference threshold;
and means for updating the background model based on the current frame. The method further comprises update rate determination means for determining an update rate based on the displacement information when the means for updating the background model updates the background model by reflecting the current frame at a predetermined update rate. 2. The object extraction device according to claim 1. 3. The object extracting apparatus according to claim 2, wherein said update rate determining means adjusts the update rate to be higher for pixels having a higher displacement speed based on said displacement information. 4. The object extracting apparatus according to claim 2, wherein said update rate determining means adjusts the update rate of pixels in which an object is detected to be low based on the result of object detection for the current frame. 3. The means for updating the background model updates the pixels identified as the foreground at a lower update rate than the pixels identified as the background based on the result of identification by the identifying means. 5. The object extracting device according to any one of 1 to 4. The means for determining the background difference threshold adjusts the background difference threshold determined based on the displacement information to be lower for pixels where objects are detected, based on results of object detection for the current frame. Item 6. An object extraction device according to any one of Items 1 to 5. 6. The object extracting apparatus according to any one of claims 2 to 5, further comprising means for initializing an update rate of each pixel whose background difference threshold has been moved based on said displacement information. 8. The object extracting apparatus according to any one of claims 1 to 7, further comprising means for initializing a background difference threshold of each pixel whose background difference threshold has been moved based on said displacement information. . In an object extraction method in which a computer extracts an object from each frame of an image by background difference calculation using a background model obtained by statistically processing past frames as a background,
detect the object area from each frame,
Get the displacement information of the object area between frames,
moving the background difference threshold used for the previous frame to the corresponding pixel region of the current frame based on the displacement information to determine the background difference threshold used for the current frame;
identifying each pixel of the current frame as either background or foreground by a background difference calculation that compares the difference between the background model and the current frame to the updated background difference threshold;
An object extraction method, comprising updating the background model based on a current frame. 10. The method of extracting an object according to claim 9, wherein an update rate for updating the background model by reflecting the current frame at a predetermined update rate is determined based on the displacement information. 11. The object extraction method according to claim 10, wherein, based on the displacement information, a pixel having a higher displacement speed is adjusted to have a higher update rate. 12. The object extraction method according to claim 10, wherein the update rate of pixels where the object is detected is adjusted to be lower based on the result of object detection for the current frame. 13. The object extraction method according to any one of claims 10 to 12, wherein pixels identified as the foreground are updated at a lower update rate than pixels identified as the background based on the result of the identification. 14. The method of any one of claims 9 to 13, wherein the background difference threshold determined based on the displacement information is adjusted lower for pixels where objects are detected based on results of object detection for the current frame. Object extraction method. An object extraction program that causes a computer to execute the object extraction method according to any one of claims 9 to 14.

Images