JP6054771B2 - Background model construction apparatus, background model construction method, and program - Google Patents

Description

  The present invention relates to a background model construction apparatus, a background model construction method, and a program for constructing a dynamic background model.

  Conventionally, a technique for extracting a dynamic foreground region such as a person or a moving object from an image taken by a fixed camera fixed at a predetermined position has been proposed.

  For example, Non-Patent Document 1 discloses a technique for constructing a background model expressed by a mixed Gaussian distribution in order to extract a dynamic foreground region in a fixed camera environment. Also, this Non-Patent Document 1 discloses a dynamic background that is robust against temporary changes in the background such as illumination fluctuations by dynamically changing the number of mixed Gaussian distributions according to the scene. Techniques for building models are also shown.

  Non-Patent Document 2 does not show a technique for constructing a dynamic background model, but shows a technique for extracting a foreground region based on parallax obtained by stereo viewing using a plurality of cameras. Has been.

Shimada, Arita, Taniguchi, “High-speed dynamic background model construction using mixed Gaussian distribution using adaptive distribution method,” IEICE Transactions Vol.J90-D, No.9, pp.2606 -2614,2007. Yuri Ivanov, Aaron Bobick, John Liu “Fast Lighting Independnt Background Subtracion” International Journal of Computer Vision (IJCV) 37 (2), pp.199-207, 2000 Chris Stauffer and W.E.L Grimson.Adaptive background mixture models for real-time tracking.Computer Vision and Pattern Recognition, Vol.2, pp.246-252, 1999. OpenCV 2.4.3, [online], [Search January 28, 2013], Internet <URL: http: //opencv.org/> Hirschmuller, H. Stereo Processing by Semiglobal Matching and Mutual Information, PAMI (30), No.2, February 2008, pp.328-341. Zhang, Z., A flexible new technique for camera calibration, IEEE JOURNALS & MAGAZINES, Volume: 22, Issue: 11, Page (s): 1330-1334, 2000.

  In the above-described technique disclosed in Non-Patent Document 1, when the foreground is moving, a dynamic background model can be constructed sequentially without confusing the foreground with the background. However, when the foreground is in a state that is almost stationary, such as when the foreground temporarily stops moving, it is difficult to essentially distinguish the foreground from the background. As a result, the foreground area has melted into the background area.

  Therefore, Non-Patent Document 1 also discloses a technique for tracking the foreground area by a method such as template matching, in addition to the construction of the background model, in order to solve the phenomenon that the foreground area merges with the background area. However, this technique for tracking the foreground region is not robust against illumination variations and foreground deformation. For this reason, if the foreground area is different from the initial template generated by template matching due to lighting fluctuations or deformation of the foreground, the foreground area is appropriately It was not possible to build a dynamic background model that could be extracted into

  The above-mentioned problem according to Non-Patent Document 1 is particularly problematic when a person's area that performs an operation that temporarily stops in the middle is extracted as a foreground area in an environment with a general background and fluctuations in illumination. Prominent. The area of a person who performs an operation that temporarily stops in the middle is, for example, an area of a person who is standing and has little movement of the lower body, but whose upper body such as the head or hand is moving.

  On the other hand, the parallax obtained by the stereo vision shown in Non-Patent Document 2 is not easily affected by illumination fluctuations. However, it is difficult to estimate parallax in an area with less texture, such as an area with few patterns or a covered area. For this reason, the technique shown in Non-Patent Document 2 cannot properly extract the foreground region when there is a region with a small amount of texture.

  Therefore, the present invention has been made in view of the above-described problems, and even when an image taken in an environment in which illumination fluctuation or foreground deformation can occur, there is a region with less texture. An object is to construct a dynamic background model that can appropriately extract the foreground region.

The present invention proposes the following matters in order to solve the above problems.
(1) The present invention is a background model construction apparatus (for example, equivalent to the background model construction apparatus 1 in FIG. 1) that constructs a dynamic background model, and includes an empty stage that does not include a foreground area based on pixel values. Area dividing means (for example, equivalent to the area dividing unit 14 in FIG. 1) and pixel information of each area obtained by dividing an image that may include a foreground area into the plurality of areas. Based on this, a background model construction device is proposed, which comprises background model update means (for example, equivalent to the background model update unit 16 in FIG. 1) for updating the background model.

  Here, an image that is a target for constructing a dynamic background model is referred to as a target image. Then, the sky stage image that does not include the foreground region is a target image that does not show the foreground, and the image that can include the foreground region is the target image.

  Therefore, according to the present invention, based on pixel values, the target image in which the foreground is not reflected is divided into a plurality of regions, and the target image is divided into the plurality of regions, based on the pixel information of each region. The background model was updated. Therefore, by dividing the target image so that pixels that are adjacent to each other and have similar pixel values are included in the same region, and refer to the pixel information of each pixel in each divided region, It can be determined whether or not it is a foreground. Therefore, even if the foreground is almost stationary, the pixel values in the foreground area can be prevented from being used for updating the background model. Therefore, it is possible to construct a dynamic background model that can suppress the foreground region from being mixed into the background region.

  (2) In the background model construction apparatus according to (1), the area dividing unit includes the image of the empty stage so that pixels adjacent to each other and having similar pixel values are included in the same area. Is divided into the plurality of regions, and a background model construction device is proposed.

  Here, it is assumed that the sky stage image is divided into a plurality of regions based on the pixel values so that pixels adjacent to each other and having similar pixel values are included in the same region. Then, it can be assumed that the pixel values of the pixels included in the same area in the sky stage image are similar to each other in the target image in a situation in which illumination fluctuations may occur unless they are in the foreground area.

  Therefore, according to the present invention, in the background model construction device of (1), an empty stage image is divided into a plurality of regions so that pixels that are adjacent to each other and have similar pixel values are included in the same region. I did it. Therefore, it is possible to determine whether or not the pixel is the foreground by determining whether or not the pixel values of the pixels of the target image included in the same region are similar.

  (3) In the background model construction device according to (1) or (2), the background model update unit includes: a median value of pixel values of a region including a target pixel to be processed; a pixel value of the target pixel; Based on the difference between the two, a background model construction apparatus is proposed in which it is determined whether or not the target pixel is a foreground, and the background model is updated.

  Here, as described above, if the sky stage image is divided so that pixels that are adjacent to each other and have similar pixel values are included in the same area, the sky stage image is divided into the same area. It can be assumed that the pixel values of the included pixels are similar to each other in the target image in a situation where illumination fluctuations may occur unless the foreground region. For this reason, it is possible to regard the median value of the pixel values of the region including the target pixel in the target image as the pixel value of the background region hidden in the foreground region.

  Therefore, according to the present invention, in the background model construction device of (1) or (2), the target pixel is based on the difference between the median pixel value of the region including the target pixel and the pixel value of the target pixel. It is determined whether or not is a foreground, and the background model is updated. For this reason, when the above-described difference is large, it is determined that the target pixel is the foreground, and when the above-described difference is small, it is determined that the target pixel is the background, thereby accurately obtaining the foreground region. Can build a dynamic background model accurately.

  (4) In the background model construction device according to (3), when the background model update unit determines that the target pixel is a foreground based on the difference, a pixel in a region including the target pixel A background model is updated using a median value, and when it is determined that the target pixel is not a foreground based on the difference, the background model is updated using the current pixel value of the target pixel. A background model construction device is proposed.

  According to the present invention, in the background model construction device of (3), when it is determined that the target pixel is the foreground based on the above-described difference, the background value is determined using the median value of the pixel values of the region including the target pixel It was decided to update the model. Therefore, the background model can be updated by using the median pixel value of the area including the target pixel in the target image as the pixel value of the background area hidden in the foreground area. Therefore, since the influence of the foreground pixel value on the background model update can be suppressed, a dynamic background model can be accurately constructed.

  According to the present invention, in the background model construction device of (3), when it is determined that the target pixel is not the foreground based on the above-described difference, the background model is determined using the current pixel value of the target pixel. I decided to update it. For this reason, the background model can be updated using the pixel value as it is for the background region, so that a dynamic background model can be accurately constructed.

  (5) The present invention relates to a stereo corresponding point search means (for example, equivalent to the stereo corresponding point search unit 21 in FIG. 5) for obtaining parallax information of a stereo image for any of the background model construction devices of (1) to (4). And the background model update unit updates the background model based on the pixel information of each region divided by the region dividing unit and the disparity information obtained by the stereo corresponding point search unit. We have proposed a background model construction device characterized by

  According to this invention, in the background model construction device according to any one of (1) to (4), the background model is updated based on the pixel information of each region obtained by dividing the image and the parallax information of the stereo image. I decided to do it. For this reason, for backgrounds with complex textures, the background model is updated based on the parallax information of the stereo image, so that the dynamic background model can be accurately identified even when illumination fluctuations or foreground deformation occur. Can be built. On the other hand, for areas with less texture, the background model is updated based on the pixel information of each area obtained by dividing the image, so that even if there are areas with less texture, Can be constructed accurately.

  (6) The present invention relates to the background model construction device of (5), depth information calculation means for obtaining depth information from the parallax information obtained by the stereo corresponding point search means (for example, in the depth information calculation unit 22 in FIG. 5). And a depth confidence value map generation means (e.g., equivalent to the depth confidence value map generation unit 23 in FIG. 5) for obtaining the reliability of the depth information obtained by the depth information calculation means, and the background model update means Whether to generate a background model based on the pixel information of each region divided by the region dividing unit based on the reliability of the depth information obtained by the depth reliability value map generating unit, or the stereo corresponding point search A background model construction apparatus is proposed that determines, for each pixel, whether a background model is to be generated based on disparity information obtained by means.

  Here, in general, depth information can be obtained from disparity information, and the reliability of depth information is high in an area where the texture is complex, and the reliability of depth information is low in an area where the texture is small.

  Therefore, according to the present invention, in the background model construction device of (5), based on the reliability of the depth information obtained from the disparity information, the background model is updated based on the pixel information of each region into which the image is divided. Whether to perform the determination based on the parallax information of the stereo image is determined for each pixel. For this reason, based on the reliability of the depth information, it can be determined whether the texture is a complex region or a region with few textures. Therefore, the background model can be updated by a method suitable for the texture.

  (7) In the background model construction apparatus according to (6), the background model update unit includes initial depth information of the target pixel obtained by the depth information calculation unit from a stereo image that does not include a foreground, and the target Based on the difference from the current depth information of the pixel, a background model construction device is proposed in which it is determined whether or not the target pixel is the foreground, and the background model is updated.

  Here, the depth information of the pixel in the foreground region and the depth information of this pixel in the video shot of the sky stage are greatly different.

  Therefore, according to the present invention, in the background model construction device of (6), it is determined whether or not the target pixel is the foreground based on the difference between the initial depth information of the target pixel and the current depth information. It was decided to update the model. For this reason, when the above-described difference is large, it is determined that the target pixel is the foreground, and when the above-described difference is small, it is determined that the target pixel is the background, thereby accurately obtaining the foreground region. Can build a dynamic background model more accurately.

  (8) In the background model construction device according to (7), the background model update unit calculates a difference between a median pixel value of a region including a target pixel to be processed and a pixel value of the target pixel. And determining whether the target pixel is a foreground, the reliability of the depth information obtained by the depth confidence value map generating means is greater than a predetermined threshold, and the target based on the difference When it is determined that the pixel is the foreground, the background model is updated using the value used for the background model update in the immediately preceding frame, and the reliability of the depth information obtained by the depth confidence value map generation unit is If it is determined that the target pixel is a foreground based on the difference, the background model is updated using the median value of the pixel value of the region including the target pixel, Difference or When the target pixel on the basis of the serial difference is determined not to be a foreground proposes a background model building apparatus characterized by updating the background model using the current pixel value of the target pixel.

  According to the present invention, in the background model construction device of (7), when the reliability of the depth information is high and it is determined that the target pixel is the foreground based on the above-described difference, the background is displayed in the immediately preceding frame. The background model was updated using the values used for updating the model. For this reason, foreground regions with complex textures, the influence on the update of the dynamic background model can be greatly suppressed, and the dynamic background model can be constructed more accurately.

  Further, according to the present invention, in the background model construction device of (7), when the reliability of the depth information is high and it is determined that the target pixel is the foreground based on the above difference, the target pixel is determined. The background model is updated using the median of the pixel values of the included region. Therefore, the background model can be updated by using the median pixel value of the area including the target pixel in the target image as the pixel value of the background area hidden in the foreground area. Therefore, since the influence of the foreground pixel value on the background model update can be suppressed, a dynamic background model can be accurately constructed.

  Also, according to the present invention, in the background model construction device of (7), when it is determined that the target pixel is not a foreground based on the above-described difference or difference, the background is generated using the current pixel value of the target pixel. It was decided to update the model. For this reason, the background model can be updated using the pixel value as it is for the background region, so that a dynamic background model can be accurately constructed.

  (9) The present invention includes an area dividing means (for example, equivalent to the area dividing section 14 in FIG. 1) and a background model updating means (for example, equivalent to the background model updating section 16 in FIG. 1), and a dynamic background model Is a background model construction method in a background model construction apparatus (e.g., equivalent to the background model construction apparatus 1 in FIG. 1), in which the region dividing unit divides an image into a plurality of regions based on pixel values. A first step (e.g., corresponding to the process of step S13 in FIG. 3) and a second in which the background model update means updates the background model based on the pixel information of each area divided by the area dividing means. (For example, corresponding to the processing of steps S24 and S25 in FIG. 4).

  According to the present invention, the image is divided into a plurality of areas based on the pixel values, and the background model is updated based on the pixel information of each divided area. For this reason, the effect similar to the effect mentioned above can be produced.

  (10) The present invention includes a region dividing unit (for example, equivalent to the region dividing unit 14 in FIG. 1) and a background model updating unit (for example, equivalent to the background model updating unit 16 in FIG. 1), and a dynamic background model Is a program for causing a computer to execute a background model construction method in a background model construction apparatus (e.g., equivalent to the background model construction apparatus 1 in FIG. 1), wherein the region dividing means is based on pixel values. The first step of dividing the image into a plurality of regions (for example, corresponding to the process of step S13 in FIG. 3) and the background model update unit are based on the pixel information of each region divided by the region dividing unit. A program for causing the computer to execute the second step of updating the background model (e.g., corresponding to steps S24 and S25 in FIG. 4). There.

  According to the present invention, by executing a program using a computer, an image is divided into a plurality of regions based on pixel values, and the background model is updated based on pixel information of each divided region. For this reason, the effect similar to the effect mentioned above can be produced.

  According to the present invention, a dynamic background that can appropriately extract a foreground area even in a case where an area photographed in an environment in which illumination fluctuations or foreground deformation can occur, includes an area with less texture. You can build a model.

It is a block diagram of the background model construction device concerning a 1st embodiment of the present invention. It is a flowchart of the background model construction process which the background model construction apparatus which concerns on the said embodiment performs. It is a flowchart of the initial background model production | generation process which the background model construction apparatus which concerns on the said embodiment performs. It is a flowchart of the background model update process which the background model construction apparatus concerning the embodiment performs. It is a block diagram of the background model construction apparatus concerning a 2nd embodiment of the present invention. It is a figure for demonstrating the search of the stereo correspondence relationship which the background model construction apparatus concerning the embodiment performs. It is a flowchart of the background model construction process which the background model construction apparatus which concerns on the said embodiment performs. It is a flowchart of the initial background model production | generation process which the background model construction apparatus which concerns on the said embodiment performs. It is a flowchart of the background model update process which the background model construction apparatus concerning the embodiment performs. It is a flowchart of the background model update process which the background model construction apparatus concerning the embodiment performs.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the constituent elements in the following embodiments can be appropriately replaced with existing constituent elements, and various variations including combinations with other existing constituent elements are possible. Accordingly, the description of the following embodiments does not limit the contents of the invention described in the claims.

<First Embodiment>
[Configuration and operation of background model construction apparatus 1]
FIG. 1 is a block diagram of a background model construction apparatus 1 according to the first embodiment of the present invention. The background model construction device 1 includes a control unit 11, an initial background model generation unit 12, a frame smoothing unit 13, a region division unit 14, a foreground candidate region extraction unit 15, and a background model update unit 16.

  The control unit 11 receives input of images and images taken by one camera, and performs various processes (for example, a process in step S5 described later). As an image for which the control unit 11 receives an input, there is an image of a scene (sky stage) that does not include a foreground. An empty stage video is a video for which a dynamic background model is to be constructed and does not show a foreground. For example, when the construction of a dynamic background model is started, the control unit 11 Entered. An image for which the control unit 11 receives an input is an image of the current frame. An image of a current frame is an image of a scene that can include a foreground, in other words, each image that constitutes a video for which a dynamic background model is to be constructed. Each time it is updated, it is input to the control unit 11.

  The initial background model generation unit 12 generates an initial background model based on pixel values. Specifically, an initial background model is constructed for each pixel as follows using the technique shown in Non-Patent Document 3, using an image of an empty stage whose input has been received by the control unit 11.

Here, paying attention to one camera image I (x, y) of the plurality of camera images constituting the above-described sky stage image, the pixel value of the pixel {x, y} at time _t is expressed as X _t. I will do it. Then, the pixel values {X ₁ , X ₂ ,... X _t } obtained up to time t can be modeled using K Gaussian distributions, and the probability of the current pixel X _t is Can be expressed by the following mathematical formula (1).

In Equation (1), w _{k, t} represents the weight of the kth Gaussian distribution at time t, μ _{k, t} represents the average value of the kth Gaussian distribution at time t, and Σ _{k, t} Represents a variance-covariance matrix of the kth Gaussian distribution at time t. K indicates the number of Gaussian distributions, which is set in advance according to the available memory area, and is generally about 3 to 5. Moreover, η is a density function of Gaussian distribution expressed by the following formula (2).

Then, the initial background model generation unit 12 first searches the K distributions for a distribution that matches the newly observed pixel value X _{t + 1 at} time t + 1. Specifically, if the pixel value _{Xt + 1} is within the standard deviation 2.5 from the average value of the distribution, it is assumed that the pixel value _{Xt + 1} matches the distribution.

  Next, when a matching distribution is found, the weights of the K Gaussian distributions and the average value and the variance value of the matching distribution are updated. On the other hand, if a matching distribution cannot be found, the distribution having a large variance and a small weight is deleted, the current pixel value is newly set as an average value, and the distribution is calculated using the deleted variance value and weight. create.

  Next, B distributions satisfying the following mathematical formula (3) are set as new background models. In Equation (3), T represents a predetermined threshold value. This mathematical formula (3) means that the weights of the Gaussian distribution are added in the descending order of matching, and the distribution when the sum of the weights exceeds the threshold T for the first time is used as the background model. .

  The frame smoothing unit 13 generates a smoothed image that averages all pixel values of the video from the empty stage video that has been input by the control unit 11.

  The area dividing unit 14 performs area dividing processing on the smoothed image generated by the frame smoothing unit 13. According to this area division processing, pixels that are adjacent to each other and have similar pixel values are classified into the same divided area, and the smoothed image is divided into a plurality of divided areas. Note that the region segmentation processing uses, for example, a program that uses an image pyramid (PyrSegmentation ()) or a Mean-Shift method (PyrMeanShiftFiltering ()) in OpenCV shown in Non-Patent Document 4. be able to.

The foreground candidate area extraction unit 15 extracts a foreground candidate area from the image of the current frame that has been input by the control unit 11. Specifically, the foreground candidate region is extracted from the image of the current frame whose input is accepted by the control unit 11 by searching for a distribution in which the pixel value _Xt matches by the technique shown in Non-Patent Document 3. To do. Whether the pixel value _Xt matches, for example, if the current pixel value is within the standard deviation 2.5 from the average value of the distribution, it is assumed that the pixel value Xt matches the distribution, and the distribution is a background model. In some cases, the pixel is set as the background, and when the distribution is not a background model, the pixel is set as the foreground candidate.

  The background model update unit 16 updates the background model based on the current frame image. Specifically, it is determined for each pixel whether or not it is truly foreground. For a pixel that is determined to be truly foreground, the current pixel value of the pixel is discarded, and the background model is updated using the median value of the divided area including the pixel. On the other hand, for the pixel determined to be the background, the background model is updated using the current pixel value of the pixel.

  The background model construction apparatus 1 having the above configuration performs background model construction processing described later with reference to FIG. 2, for each pixel, generation of an initial background model that is an initial background model, update of the background model, I do.

  FIG. 2 is a flowchart of background model construction processing performed by the background model construction device 1.

  In step S <b> 1, the background model construction device 1 receives an input of an empty stage video by the control unit 11. When an empty stage video is input, the process proceeds to step S2.

  In step S2, the background model construction device 1 performs an initial background model generation process to generate an initial background model, and moves the process to step S3. Details of the initial background model generation processing performed by the background model construction device 1 will be described later with reference to FIG.

  In step S <b> 3, the background model construction device 1 accepts input of an image of the current frame by the control unit 11. When the current frame image is input, the process proceeds to step S4.

  In step S4, the background model construction device 1 performs a background model update process, updates the background model based on the image of the current frame, and moves the process to step S5. Details of the background model update processing performed by the background model construction device 1 will be described later with reference to FIG.

  In step S5, the background model construction device 1 determines whether or not the background model has been updated up to the final frame by the control unit 11. If it is determined that it has been performed, the background model construction process shown in FIG. 2 is terminated. If it is determined that it has not been performed, the process proceeds to step S6.

  In step S6, the background model construction device 1 causes the control unit 11 to update the processing target frame that is a frame for performing the processes in steps S3 to S5, and returns the process to step S3.

  FIG. 3 is a flowchart of the initial background model generation process performed by the background model construction apparatus 1.

  In step S11, the background model construction device 1 uses the initial stage model generation unit 12 to generate the initial background model based on the pixel values using the empty stage image received in step S1, and then proceeds to step S12. Move processing.

  In step S12, the background model construction device 1 uses the frame smoothing unit 13 to generate a smoothed image that averages all the pixel values of this video from the empty stage video that has been input in step S1, and then in step S13. Move processing to.

  In step S13, the background model construction device 1 causes the region dividing unit 14 to perform region division processing on the smoothed image generated in step S12, and ends the initial background model generation processing illustrated in FIG.

  FIG. 4 is a flowchart of the background model update process performed by the background model construction device 1.

  In step S21, the background model construction device 1 uses the foreground candidate area extraction unit 15 to extract a foreground candidate area from the current frame image received in step S3, and moves the process to step S22.

  In step S22, the background model construction device 1 determines whether each pixel is a foreground candidate based on the result of extracting the foreground candidate region in step S21 by the background model update unit 16. If it is determined that it is a foreground candidate, the process proceeds to step S23, and if it is determined that it is not a foreground candidate, the process proceeds to step S25.

  In step S23, the background model construction device 1 uses the background model update unit 16 to divide the image of the current frame into a plurality of divided areas in step S13, and in step S22, foreground For each pixel determined to be a candidate, it is determined whether it is truly a foreground. Specifically, first, the image of the current frame is divided into a plurality of divided regions divided in step S13. Next, for each pixel determined to be a foreground candidate in step S22, the median value of the pixel values of the divided region including the pixel (hereinafter referred to as the target pixel) among the divided regions of the current frame image, The difference between the pixel value of the target pixel is obtained. Next, when the obtained difference is larger than a predetermined threshold, it is determined that the target pixel is truly a foreground, and the process proceeds to step S24. On the other hand, when the obtained difference is equal to or less than a predetermined threshold, it is determined that the target pixel is the background, and the process proceeds to step S25.

  In step S24, the background model construction device 1 causes the background model update unit 16 to discard the current pixel value of the target pixel and update the background model using the median value of the pixel values of the divided area including the target pixel. Then, the background model update process shown in FIG.

  In step S25, the background model construction device 1 uses the background model update unit 16 to update the background model using the current pixel value of the target pixel, and ends the background model update process illustrated in FIG.

  According to the background model construction apparatus 1 described above, the following effects can be achieved.

  The background model construction device 1 divides the sky stage image into a plurality of regions based on the pixel values so that pixels that are adjacent to each other and have similar pixel values are included in the same region. For this reason, the pixel values of the pixels included in the same area in the sky stage image are similar to those in the current frame image in a situation where illumination fluctuations may occur unless they are in the foreground area. Can be assumed. For this reason, it is possible to regard the median value of the pixel value of the area including the target pixel in the image of the current frame as the pixel value of the background area hidden in the foreground area.

  Therefore, the background model construction device 1 determines whether the target pixel is the foreground based on the difference between the median pixel value of the divided region including the target pixel and the pixel value of the target pixel. For this reason, the foreground region can be accurately obtained, and a dynamic background model can be accurately constructed.

  Further, the background model construction device 1 updates the background model for the pixel determined to be the foreground using the median value of the pixel values of the divided area including this pixel. Therefore, the background model can be updated using the median pixel value of the area including the target pixel in the image of the current frame as the pixel value of the background area hidden in the foreground area. Therefore, since the influence of the foreground pixel value on the background model update can be suppressed, it is possible to construct a dynamic background model that can suppress the foreground region from being mixed into the background region.

  Also, the background model construction device 1 updates the background model for the pixel determined to be the background using the current pixel value of this pixel. For this reason, the background model can be updated using the pixel value as it is for the background region, so that a dynamic background model can be accurately constructed.

Second Embodiment
[Configuration and Operation of Background Model Construction Device 1A]
FIG. 1 is a block diagram of a background model construction apparatus 1A according to the second embodiment of the present invention. The background model construction device 1A is different from the background model construction device 1 according to the first embodiment of the present invention shown in FIG. 1 in that a control unit 11A is provided instead of the control unit 11, and an initial background model generation unit 12 Instead, a point including an initial background model generation unit 12A, a point including a frame smoothing unit 13A instead of the frame smoothing unit 13, a point including a background model updating unit 16A instead of the background model updating unit 16, and a stereo It differs from the point provided with the corresponding point search part 21, the depth information calculation part 22, and the depth reliability value map generation part 23. In the background model construction device 1A, the same constituent elements as those of the background model construction device 1 are denoted by the same reference numerals, and the description thereof is omitted.

  The control unit 11A accepts input of images and images taken by each of the two cameras, and performs various processes (for example, a process in step S55 described later). The control unit 11A accepts input as an image of an empty stage (hereinafter referred to as an L image of the empty stage) taken with the left camera out of the two cameras and an image of the two cameras. There is an image of an empty stage (hereinafter referred to as an R image of the empty stage) taken with the one arranged on the right side. The empty stage L video and R video are input to the control unit 11A when, for example, construction of a dynamic background model is started. The control unit 11A accepts an input as an image of the current frame (hereinafter referred to as an L image of the current frame) captured by the left camera of the two cameras, and two images. There is an image of the current frame (hereinafter referred to as an R image of the current frame) captured by the camera arranged on the right side of the camera. The L image and R image of the current frame are images of a scene that can include the foreground, and are input to the control unit 11A each time the frame is updated in a period in which the background model is updated, for example.

  Similar to the initial background model generation unit 12, the initial background model generation unit 12A generates an initial background model based on pixel values. However, the initial background model generation unit 12 generates an initial background model using an empty stage image captured by one camera, whereas the initial background model generation unit 12A captures images by two cameras. An initial background model is generated using the image of the sky stage (L image and R image of the sky stage).

  Similar to the frame smoothing unit 13, the frame smoothing unit 13 </ b> A generates a smoothed image. However, the frame smoothing unit 13 generates a smoothed image from an image of an empty stage photographed by one camera, whereas the frame smoothing unit 13A is an empty stage photographed by two cameras. Smoothed images are generated from each of the videos (L video and R video of the empty stage). For this reason, two types of smoothed images are generated in the frame smoothing unit 13A.

  The stereo correspondence point search unit 21 searches for stereo correspondence. The search for the stereo correspondence is performed for each pixel of the smoothed image of the L-stage of the empty stage and each pixel of the smoothed image of the R-picture of the empty stage (see FIG. 6), and the current There are cases where it is performed on each pixel of the L image of the frame and each pixel of the R image of the current frame. In the stereo correspondence search, the stereo correspondence point search unit 21 uses a block matching method that takes into consideration consistency in a wide area shown in Non-Patent Document 5.

  In the search for the stereo correspondence, the stereo correspondence point search unit 21 performs a luminance-based correspondence point search. In this case, OpenSG V's StereoSGBM () shown in Non-Patent Document 4 can be used.

  In addition, in order to search for stereo correspondence, the stereo correspondence point search unit 21 uses two known plane objects such as a chess board in advance based on the technique disclosed in Non-Patent Document 6. The internal parameters and external parameters of each of the cameras are acquired, and a left parallel stereo image corresponding to the left camera and a right parallel stereo image corresponding to the right camera are created. Specifically, it can be realized by using stereoRectify () and remap () of the open source library OpenCV shown in Non-Patent Document 4.

  The depth information calculation unit 22 uses the left parallel stereo image and the right parallel stereo image created by the stereo corresponding point search unit 21 and uses the left parallel stereo image and the depth information of each point indicated by each pixel of the left and right parallel stereo images. Ask for.

Here, the left parallel stereo image is represented as I _l (x, y), and the right parallel stereo image is represented as I _r (x, y). Then, a point corresponding to an arbitrary point I _l (x _l , y _l ) in the left parallel stereo image I _l exists on I _r (x _r , y _r ) in the right parallel stereo image I _r. Will do. It should be _noted, is a _y l = y _r. Therefore, assuming that the point of the right parallel stereo image corresponding to the point I _l (x _l , y) of the left parallel stereo image is the point I _r (x _r , y), from these parallel stereo images, Depth information Z shown in the following mathematical formula (4) can be obtained.

In Equation (4), c represents the baseline length, f represents the focal length, and d represents the parallax obtained from x ₁ −x _r .

  The depth confidence value map generation unit 23 uses a cost function (for example, defined by the technique shown in Non-Patent Document 6) in the stereo correspondence point search by the stereo correspondence point search unit 21 to determine the reliability of block matching. Estimation is performed for each pixel, and the reliability of the depth information obtained by the depth information calculation unit 22 is estimated for each pixel. The reliability of block matching can be estimated from the difference between the minimum value of the cost function described above and the second smallest value of the cost function described above.

  The background model update unit 16A updates the background model based on the current frame image. Specifically, it is determined for each pixel whether or not it is truly foreground. Then, the reliability of the depth information estimated by the depth confidence value map generation unit 23 is high, and for the pixel determined to be the foreground, the background model is updated using the value used for the background model update in the immediately preceding frame. To do. On the other hand, for the pixel whose depth information estimated by the depth confidence value map generation unit 23 has low reliability and is determined to be a foreground, the current pixel value of the pixel is discarded, and the center of the divided region including the pixel is discarded. Update the background model with the value. In other cases, the background model is updated using the current pixel value of the pixel.

  The background model construction device 1A having the above configuration performs background model construction processing described later with reference to FIG. 7, for each pixel, generation of an initial background model that is an initial background model, update of the background model, I do.

  FIG. 7 is a flowchart of background model construction processing performed by the background model construction device 1A.

  In step S51, the background model construction device 1A accepts input of an L video and an R video of the empty stage by the control unit 11A. Then, when the L video and R video of the empty stage are input, the process proceeds to step S52.

  In step S52, the background model construction device 1A performs initial background model generation processing to generate an initial background model, and moves the processing to step S53. Details of the initial background model generation processing performed by the background model construction device 1A will be described later with reference to FIG.

  In step S53, the background model construction device 1A accepts input of the L image and the R image of the current frame by the control unit 11A. When the L image and R image of the current frame are input, the process proceeds to step S54.

  In step S54, the background model construction device 1A performs background model update processing, updates the background model based on the image of the current frame, and moves the processing to step S55. Details of the background model update processing performed by the background model construction device 1A will be described later with reference to FIGS.

    In step S55, the background model construction device 1A determines whether or not the background model has been updated up to the final frame by the control unit 11A. If it is determined that it has been performed, the background model construction process shown in FIG. 7 is terminated. If it is determined that it has not been performed, the process proceeds to step S56.

  In step S56, the background model construction device 1A updates the processing target frame, which is a frame for performing the processes in steps S53 to S55, by the control unit 11A, and returns the process to step S53.

  FIG. 8 is a flowchart of the initial background model generation process performed by the background model construction device 1A.

  In step S61, the background model construction device 1A generates an initial background model based on the pixel values by using the L-stage and R-pictures of the empty stage received in step S51 by the initial background model generation unit 12A. The process proceeds to step S62.

  In step S62, the background model construction device 1A uses the frame smoothing unit 13A to perform smoothing by averaging all pixel values of the sky stage L video from the sky stage L video and the R video received in step S51. An image and a smoothed image obtained by averaging all pixel values of the R video of the sky stage are generated, and the process proceeds to step S63.

  In step S63, the background model construction device 1A uses the stereo corresponding point search unit 21 to search for a stereo correspondence relationship between the L video and the R video of the empty stage, and moves the process to step S64.

  In step S64, the background model construction device 1A obtains initial depth information by the depth information calculation unit 22, and the process proceeds to step S65. Specifically, using the left parallel stereo image and the right parallel stereo image generated in step S63, the depth value of each point indicated by each pixel of the left and right parallel stereo images is obtained, and the initial depth information and To do.

  In step S65, the background model construction device 1A obtains the reliability of the initial depth information obtained in step S64 for each pixel by the depth confidence value map generation unit 23, and moves the process to step S66.

  In step S66, the background model construction device 1A causes the region dividing unit 14 to perform region division processing on the smoothed image generated in step S62, and ends the initial background model generation processing illustrated in FIG.

  9 and 10 are flowcharts of the background model update process performed by the background model construction apparatus 1A.

  In step S71, the background model construction device 1A causes the stereo correspondence point search unit 21 to search for the stereo correspondence relationship between the L image and the R image of the current frame received in step S53, and then proceeds to step S72. Move processing.

  In step S72, the background model construction device 1A obtains the depth information from the L image and the R image of the current frame received in step S53 by the depth information calculation unit 22, and moves the process to step S73.

  In step S73, the background model construction device 1A causes the foreground candidate area extraction unit 15 to extract the foreground candidate areas from the L image and R image of the current frame received in step S53, and the process proceeds to step S74.

  In step S74, the background model construction device 1A determines whether each pixel is a foreground candidate based on the result of extracting the foreground candidate area in step S73 by the background model update unit 16A. If it is determined that it is a foreground candidate, the process proceeds to step S76. If it is determined that it is not a foreground candidate, the process proceeds to step S75.

  In step S75, the background model construction device 1A causes the background model update unit 16A to update the background model using the current pixel value of the target pixel, and ends the background model update process illustrated in FIGS.

  In step S76, the background model construction device 1A determines whether the initial depth confidence value obtained in step S65 is greater than a predetermined threshold value for each pixel determined to be a foreground candidate in step S74 by the background model update unit 16A. Determine whether or not. If it is larger, the process proceeds to step S79. If it is not larger, the process proceeds to step S77.

  In step S77, the background model construction device 1A, for each pixel determined to be a foreground candidate in step S74 by the background model update unit 16A, the median value of the pixel values of the divided region including the target pixel and the pixel of the target pixel The difference between the value and the value is obtained. If the obtained difference is larger than a predetermined threshold, it is determined that the target pixel is truly a foreground, and the process proceeds to step S78. On the other hand, when the obtained difference is equal to or less than a predetermined threshold, it is determined that the target pixel is the background, and the process proceeds to step S75.

  In step S78, the background model construction device 1A causes the background model update unit 16A to discard the current pixel value of the target pixel and update the background model using the median value of the pixel values of the divided region including the target pixel. The background model update process shown in FIGS.

  In step S79, the background model construction device 1A has a value obtained by subtracting the current depth value from the initial depth information obtained in step S64 for each pixel determined to be a foreground candidate in step S74 by the background model update unit 16A. It is determined whether or not it is larger than a predetermined threshold value. If it is larger, it is determined that the target pixel is truly the foreground, and the process proceeds to step S80. On the other hand, if it is not large, it is determined that the target pixel is the background, and the process proceeds to step S75.

  In step S80, the background model construction device 1A causes the background model update unit 16A to discard the current pixel value of the target pixel, and use the value used to update the background model in the immediately previous frame, to The background model update process shown in FIGS. When the current frame is the first frame, the background model is updated using the average pixel value at the time of shooting the empty stage.

  According to the above background model construction device 1A, the following effects can be obtained.

  The background model construction device 1A obtains depth information using the parallax information of the left and right parallel stereo images for an area having a complex texture, and updates the background model based on the depth information. For this reason, a dynamic background model can be accurately constructed even when illumination variation or foreground deformation occurs.

  Further, the background model construction device 1A updates the background model based on the pixel information of each divided region obtained by dividing the image for a region with less texture. For this reason, a dynamic background model can be accurately constructed even when a region with a small amount of texture exists.

  Further, the background model construction device 1A obtains the reliability of the depth information, and determines whether the texture is a complex area or an area with few textures based on the reliability of the depth information. Therefore, the background model can be updated by a method suitable for the texture.

  Further, the background model construction device 1A determines whether or not the target pixel is the foreground based on the difference between the initial depth information of the target pixel and the current depth information. For this reason, the foreground region can be accurately obtained, and a dynamic background model can be constructed more accurately.

  Further, the background model construction device 1A uses the value used for updating the background model in the immediately preceding frame for pixels for which the reliability of the depth information is high and the target pixel is determined to be the foreground based on the above-described difference. Update the background model using. For this reason, foreground regions with complex textures, the influence on the update of the dynamic background model can be greatly suppressed, and the dynamic background model can be constructed more accurately.

  Similarly to the background model construction device 1, the background model construction device 1A divides an empty stage image into a plurality of regions based on pixel values, and pixels that are adjacent to each other and have similar pixel values. Divide it so that it is included in the same area. For this reason, the pixel values of the pixels included in the same area in the sky stage image are similar to those in the current frame image in a situation where illumination fluctuations may occur unless they are in the foreground area. Can be assumed. For this reason, it is possible to regard the median value of the pixel value of the area including the target pixel in the image of the current frame as the pixel value of the background area hidden in the foreground area.

  Therefore, the background model construction device 1A determines the median value of the pixel values of the region including the target pixel for pixels for which the reliability of the depth information is high and the target pixel is determined to be the foreground based on the above-described difference. Use to update the background model. Therefore, the background model can be updated using the median pixel value of the area including the target pixel in the image of the current frame as the pixel value of the background area hidden in the foreground area. Therefore, since the influence of the foreground pixel value on the background model update can be suppressed, a dynamic background model can be accurately constructed.

  Note that the processing of the background model construction device 1 and the background model construction device 1A of the present invention is recorded on a computer-readable non-transitory recording medium, and the program recorded on this recording medium is stored in the background model construction device 1 and the background. The present invention can be realized by being read and executed by the model construction apparatus 1A.

  Here, for example, a nonvolatile memory such as an EPROM or a flash memory, a magnetic disk such as a hard disk, a CD-ROM, or the like can be applied to the above-described recording medium. Further, reading and execution of the program recorded on the recording medium is performed by a processor provided in the background model construction device 1 or the background model construction device 1A.

  The above-described program is transmitted from the background model construction device 1 or background model construction device 1A storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. May be. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  Further, the above-described program may be for realizing a part of the above-described function. Furthermore, what can implement | achieve the above-mentioned function in combination with the program already recorded on the background model construction apparatus 1 and the background model construction apparatus 1A, what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes a design that does not depart from the gist of the present invention.

  For example, in the second embodiment described above, the process in step S66 in FIG. 8 is performed after the process in step S65. However, the present invention is not limited to this, and any time after the process in step S62 is completed. Also good.

DESCRIPTION OF SYMBOLS 1, 1A ... Background model construction apparatus 11, 11A ... Control part 12, 12A ... Initial background model generation part 13, 13A ... Frame smoothing part 14 ... Area division part 15 ... Foreground candidate area extraction unit 16, 16A ... background model update unit 21 ... stereo corresponding point search unit 22 ... depth information calculation unit 23 ... depth confidence value map generation unit

Claims (9)

A background model construction device for constructing a dynamic background model,
Area dividing means for dividing an empty stage image not including the foreground area into a plurality of areas based on the pixel values;
Background model update means for updating a background model based on pixel information of each area obtained by dividing an image that may include a foreground area into the plurality of areas;
Equipped with a,
The area dividing unit divides the sky stage image into the plurality of areas so that pixels adjacent to each other and having similar pixel values are included in the same area. .   The background model update means determines whether the target pixel is a foreground based on a difference between a median pixel value of a region including the target pixel to be processed and a pixel value of the target pixel. The background model construction apparatus according to claim 1, wherein the background model is updated. The background model update means includes
If it is determined that the target pixel is the foreground based on the difference, the background model is updated using the median of the pixel values of the region including the target pixel,
The background model construction apparatus according to claim 2 , wherein when the target pixel is determined not to be a foreground based on the difference, the background model is updated using a current pixel value of the target pixel. Stereo corresponding point search means for obtaining parallax information of a stereo image is provided,
The background model update means updates the background model based on the pixel information of each area divided by the area dividing means and the disparity information obtained by the stereo corresponding point search means. The background model construction apparatus according to any one of claims 1 to 3 . Depth information calculation means for obtaining depth information from the parallax information obtained by the stereo corresponding point search means;
A depth confidence value map generating means for determining the reliability of the depth information obtained by the depth information calculating means,
Whether the background model update means generates a background model based on pixel information of each area divided by the area dividing means based on the reliability of the depth information obtained by the depth confidence value map generation means, 5. The background model construction apparatus according to claim 4 , wherein whether to generate a background model based on the disparity information obtained by the stereo corresponding point search unit is determined for each pixel. The background model update means is based on the difference between the initial depth information of the target pixel obtained by the depth information calculation means from the stereo image not including the foreground and the current depth information of the target pixel. The background model construction apparatus according to claim 5 , wherein the background model is updated by determining whether or not is a foreground. The background model update means includes
Based on the difference between the median pixel value of the region including the target pixel to be processed and the pixel value of the target pixel, it is determined whether the target pixel is a foreground,
When the reliability of the depth information obtained by the depth confidence value map generation means is greater than a predetermined threshold value, and when it is determined that the target pixel is a foreground based on the difference, the background in the immediately preceding frame Update the background model with the values used to update the model,
If the reliability of the depth information obtained by the depth confidence value map generation means is equal to or lower than a predetermined threshold value and the target pixel is determined to be a foreground based on the difference, the target pixel is included. Update the background model with the median pixel value of the region,
The background model according to claim 6 , wherein when it is determined that the target pixel is not a foreground based on the difference or the difference, a background model is updated using a current pixel value of the target pixel. Construction device. A background model construction method in a background model construction device that comprises a region division means and a background model update means, and constructs a dynamic background model,
A first step in which the area dividing means divides the image into a plurality of areas based on pixel values;
A second step in which the background model updating means updates a background model based on pixel information of each area divided by the area dividing means;
Equipped with a,
In the first step, an empty stage image is divided into the plurality of regions so that pixels adjacent to each other and having similar pixel values are included in the same region. . A program for causing a computer to execute a background model construction method in a background model construction apparatus that comprises a region division means and a background model update means, and constructs a dynamic background model,
A first step in which the area dividing means divides the image into a plurality of areas based on pixel values;
A second step in which the background model updating means updates a background model based on pixel information of each area divided by the area dividing means;
Equipped with a,
In the first step, an image of an empty stage is divided into the plurality of regions so that pixels adjacent to each other and having similar pixel values are included in the same region . Program for.

Images