JP5505936B2 - Image processing unit and image processing program - Google Patents

Description

  The present invention relates to an image processing unit that processes a frame image of an imaging area captured by an imaging device such as a video camera and detects an object captured in the frame image, and an image processing program.

  2. Description of the Related Art Conventionally, a monitoring system for monitoring the intrusion of a suspicious person or leaving a suspicious object using an imaging device such as a video camera has been put into practical use. In this type of monitoring system, the imaging area of the imaging device is matched with the monitoring target area for monitoring the intrusion of a suspicious person or the leaving of a suspicious object. In addition, an image processing unit that processes a frame image of the monitoring target area imaged by the imaging device and detects an imaged object (suspicious person, suspicious object, or the like) is provided. The host device performs processing such as notifying the staff of object detection and displaying a frame image in which the detected object is imaged on the display unit according to the object detection result in the image processing unit. .

  For each pixel of the input frame image, the image processing unit uses the background model to determine whether it is a background pixel in which a background is imaged or a foreground pixel in which an object that is not a background is imaged. Further, based on this determination result, a binarized image is generated by distinguishing between a background area where the background is imaged and a foreground area where the object is imaged. The foreground region in the binarized image is a region where an object is imaged, and the type of the object is estimated from its size and shape.

  The image processing unit uses the frame image captured by the imaging device in order to make the background model used for generating the binarized image described above correspond to environmental changes such as changes in brightness in the monitoring target area. It has been updated. For this reason, when an object is imaged in the frame image, pixels located in the foreground area where the object is imaged are updated to a background model affected by the imaged object. As a result, the object is erroneously detected as the background, and conversely, the background is erroneously detected as the object.

  Therefore, in the frame image, for the image region where the object is highly likely to be stationary, the learning rate is made smaller than other image regions (the update frequency is reduced) in order to reduce the influence on the background model to be created. A method, a method using two background models having different learning rates, and the like have been proposed (see Non-Patent Documents 1 and 2).

Arslan Basharat, Alexei Gritai, Mubarak Shah.Learning object motion patterns for anomaly detection and improved object detection.Computer Vision and Pattern Recognition, pp1-8,2008. Fatih Porikli, Yuri Ivanov, Tetsugi Haga. Robust adandoned object detection using durl foregrounds. EURASIP Journal on Advances in Signal Processing, 2008.

  However, in the conventional methods proposed in Non-Patent Documents 1 and 2, etc., when an object that is not a background captured in a frame image is stationary, it is updated to a background model that regards this object as a background. Is just extending the time. Therefore, when the time during which the object is stationary becomes longer than the time until the object is updated to the background model that regards the object as the background, the object is updated to the background model affected by the object being imaged.

  In addition, in the frame image, it may be possible to limit the area where the object is captured so that the background model is not updated. However, in this case, the background model is not updated for this area while a stationary object is being imaged. Therefore, it is impossible to cope with environmental changes such as brightness changes in the monitoring target area during this period, and when the object moves, a new object is imaged even though the background is imaged. May be falsely detected.

  An object of the present invention is to suppress the influence of an imaged object on an input frame image, and to appropriately update a model according to environmental changes such as a change in brightness in an imaging area. An object of the present invention is to provide an image processing unit and an image processing program that improve the detection accuracy of an object captured in an image.

  The image processing unit of the present invention is configured as follows in order to solve the above problems and achieve the object.

  A frame image of an imaging area captured by an imaging device such as a video camera is input to the image input unit. The probability model storage unit stores, for each pixel of the frame image, a probability model obtained by modeling the frequency of the pixel value of the pixel in the past. This probability model is, for example, a Gaussian density function. Further, the prediction model storage unit stores, for each pixel of the frame image, a prediction model obtained by modeling a time series transition of the pixel value of the pixel. This prediction model is, for example, a model based on the exponential smoothing method.

  The binarized image generation unit captures the background for each pixel of the frame image input to the image input unit based on the probability model of the corresponding pixel stored in the probability model storage unit. It is determined whether it is a background pixel or a foreground pixel in which an object that is not a background is imaged, and a binarized image to which the determination result is assigned is generated. The object detection unit detects an object located in the imaging area from the binarized image generated by the binarized image generation unit.

  Further, for the pixel determined by the binarized image generation unit as the background pixel, the model update unit updates the probability model and the prediction model for the pixel using the pixel value of the pixel. Therefore, for the background pixel, the probability model and the prediction model can be updated appropriately. In addition, for the pixel determined by the binarized image generation unit to be the foreground pixel, the model update unit is similar to the pixel, the probability model, and the prediction model, and the binarized image generation unit is the background pixel. Using the pixel value of the pixel determined to be, the probability model and the prediction model related to the pixel are updated. Therefore, the probabilistic model and the prediction model corresponding to the environmental change such as the brightness change in the monitoring target area can also be updated for the pixels in the foreground area where the object is captured in the frame image. Thereby, the detection accuracy of the object imaged in the frame image can be improved.

  The binarized image generation unit stores, for each pixel of the frame image input to the image input unit, the probability model of the corresponding pixel stored in the probability model storage unit and the prediction model storage unit. A configuration may be adopted in which it is determined whether the pixel is a background pixel or a foreground pixel based on a prediction model of the corresponding pixel. In this case, for example, the binarized image generation unit obtains the first likelihood that is the background pixel based on the probability model of the corresponding pixel stored in the probability model storage unit for each pixel of the frame image. The second likelihood which is the background pixel is calculated based on the prediction model of the corresponding pixel which is calculated and stored in the prediction model storage unit, and the calculated first likelihood and second likelihood are calculated. What is necessary is just to make it the structure which integrates and determines whether it is a background pixel or a foreground pixel.

  According to the present invention, the influence of the object being captured on the input frame image is suppressed, and the model can be updated appropriately according to environmental changes such as changes in brightness within the imaging area. It is possible to improve the detection accuracy of an object being imaged.

It is a block diagram which shows the structure of the principal part of an image processing unit. It is a flowchart which shows operation | movement of an image processing unit. It is a figure explaining the pixel number which shows each pixel of a frame image. It is a figure explaining the method of determining a foreground area | region. It is a figure explaining the update of the probability model concerning each pixel of a frame image, and a prediction model.

  Hereinafter, an image processing unit according to an embodiment of the present invention will be described.

  FIG. 1 is a block diagram showing the configuration of the main part of the image processing unit. The image processing unit 1 includes a control unit 2, an image input unit 3, an image processing unit 4, and an input / output unit 5.

  The control unit 2 controls the operation of each part of the main body of the image processing unit 1.

  The image input unit 3 receives a frame image captured by the video camera 10. For example, the video camera 10 outputs 30 frame images per second. The video camera 10 is installed so that a monitoring target area for monitoring the intrusion of a suspicious person, the leaving of a suspicious object, or the like is within the imaging area.

  The image processing unit 4 processes the frame image input to the image input unit 3 and detects an object (non-background object) captured in the frame image. In addition, the image processing unit 4 performs model update processing for updating a probability model and a prediction model described later using the frame image input to the image input unit 3. The image processing unit 4 includes a memory 4a that stores a probability model and a prediction model. The memory 4a corresponds to the probability model storage unit and the prediction model storage unit referred to in the present invention.

  The image processing unit 4 includes a microcomputer that executes image processing on the frame image input to the image input unit 3. The image processing program according to the present invention is a program for operating this microcomputer and is installed in the image processing unit 4 in advance.

  The input / output unit 5 controls data input / output with a host device (not shown). When the image processing unit 4 detects an object, the input / output unit 5 outputs a signal for notifying a higher-level device (not shown) to that effect. The input / output unit 5 may be configured to output a signal notifying the detection of the object, or may be configured to output a signal notifying the detection of the object together with the frame image in which the object is detected.

  When the host device is notified of the detection of an object located in the monitoring target area by a signal output from the input / output unit 5 of the image processing unit 1, the host device outputs a message to that effect and It may be configured to notify a staff member or the like in the office, or may be configured to display the frame image on a display when a frame image in which an object is detected is transmitted.

  Next, the operation of the image processing unit 1 will be described. FIG. 2 is a flowchart showing the operation of the image processing unit. First, an outline of the operation of the image processing unit will be described, and then each operation will be described in detail.

  When the frame image captured by the video camera 10 is input to the image input unit 3 (s1), the image processing unit 1 captures the input frame image. The image processing unit 4 calculates the background likelihood (first likelihood referred to in the present invention) with the probability model stored in the memory 4a for each pixel of the frame image (s2). Further, the image processing unit 4 calculates the background likelihood (second likelihood referred to in the present invention) with the prediction model stored in the memory 4a for each pixel of the frame image input this time (s3).

  The probability model is obtained by modeling the occurrence frequency of pixel values in a frame image input in the past using a mixed Gaussian distribution for each pixel of the frame image. The prediction model models a time-series transition of pixel values in a frame image input in the past for each pixel of the frame image. Therefore, the prediction model can be used for estimating the pixel value in the next input frame image.

  The image processing unit 4 captures the background of the frame image input this time using the background likelihood in the probability model calculated in s2 for each pixel and the background likelihood in the prediction model calculated in s3. It is determined whether it is a pixel (hereinafter referred to as background pixel) or a pixel in which an object that is not the background is imaged (hereinafter referred to as foreground pixel) (s4). In s4, an energy function based on a Markov random field is defined, and the minimization problem is solved using a graph cut to determine and determine whether the pixel is a background pixel or a foreground pixel. By the processing according to s4, a binary image is generated by dividing a background area where the background is imaged and a foreground area where the foreground (object) is imaged in the frame image. The image processing unit 4 performs an object detection process for estimating the type or the like of the imaged object from the size and shape of the foreground area in the binarized image (s5).

Further, the image processing unit 4 selects a pixel for updating the probability model and the prediction model for each pixel of the frame image (s6). For each pixel of the frame image, the image processing unit 4 uses the pixel value of the pixel selected in s6 to update the probability model and the prediction model stored in the memory 4a for that pixel (s7). In s6, for the pixel determined to be the background, that pixel is selected. For a pixel determined to be a foreground, a pixel that is determined to be a background pixel this time and that has a similar probability model and prediction model is selected from the peripheral pixels of the pixel.

  In general, the pixel value of the background pixel changes periodically with the shaking of the plant. Also, the brightness due to changes in sunshine or the like does not change rapidly but changes gradually.

  Hereinafter, the operation of s2 to s7 described above will be described in detail.

Here, the frame image input to the image input unit 3 is an image of n pixels in the main scanning direction (left-right direction in FIG. 3) and m pixels in the sub-scanning direction (up-down direction in FIG. 3) as shown in FIG. Will be described. Also, the pixel number i of the pixel that is Nth in the main scanning direction and Mth in the subscanning direction is
i = n × (M−1) + N
And Further, at time t, the pixel value of each pixel i of the frame image input to the image input unit 3 is represented by ^x i ^t .

  In the following description, when describing an arbitrary pixel, the subscript i may be omitted.

For pixel i, when the k-th Gaussian distribution at time t has weight wk ^t , mean value μk ^t , and variance-covariance matrix Σk ^t , the background likelihood by the probability model is

Calculate with Where η is

It is a density function of Gaussian distribution expressed by

Update of the probabilistic model, the pixel value x ^t, look for a matching distribution from the K distribution. If the pixel value x ^t is within the average value γδ of each distribution, it is determined that the distribution matches. δ is a standard deviation of this distribution, and γ may be set to γ = 2.5, for example.

Also, wk ^t is the weight of K Gaussian distributions.

Update with. In the equation (3), α is a learning rate, and when this value is large, the weight of the distribution that matches the new observed pixel value x ^t immediately increases. That is, a new probability model is constructed at a relatively early stage.

In addition, Mk ^t is, for the matched distribution 1, to any other distribution to 0. After updating the weight of each distribution, the image processing unit 4 normalizes the sum of the weights to be 1.

If a matching distribution is found, the average μ ^{t of the} distribution is

And the variance value δ ^t

Update with. Ρ in the equations (4) and (5) is a learning rate,

It is represented by

Further, y ^t is the pixel value to update the probabilistic model, as described below, the pixels determined as the foreground, since different pixels determined as the background, here in the notation distinguished from x ^t .

  Further, when no matching distribution is found, the image processing unit 4 newly creates a Gaussian distribution ηk + 1. here,

And W is a weight set in advance for a new distribution.

  Further, the image processing unit 4 calculates the matching degree w / δ of the Gaussian distribution. If the weight of the Gaussian distribution having the smallest matching degree is equal to or less than a predetermined threshold, the image processing unit 4 deletes the distribution, and the remaining Normalization is performed so that the sum of weights of the distribution is 1.

  Next, the prediction model will be described. The prediction model of the observed pixel value of each pixel of the frame image input to the image input unit 3 is updated by an exponential smoothing method. As is well known, the exponential smoothing method is a time series analysis technique used to predict future values from time series data. Here, it is used to model the trend of past observed pixel values and to predict the observed pixel value at the next time.

A constant model that sequentially smoothes time-series data using the moving average value m ^t and the observed value yt at time ^t is:

It is represented by Here, β (0 <β <1) is a smoothing coefficient, and the closer the β is to 1, the more important the prediction is the previous value.

If the case of a constant model is not prone to time-series data, the following are shown (9), but m ^t is the effective prediction value at time t + 1, which increases the data, or there is a downward trend, the The trend cannot be predicted. Here, in consideration of such a tendency, a linear trend model is used. The linear model is introduced estimate r ^t trend in time t, the predicted value z ^t of linear trend model,

Can be predicted.

  The background likelihood evaluation using the above prediction model will be described. When the above prediction model is used in units of pixels, there is a high possibility that a prediction error will occur. Here, a target pixel (prediction target pixel) and a plurality of pixels located around the target pixel are used, and a space is used. The background likelihood is calculated in consideration of typical information.

Specifically, let R be a set of the target pixel i and four neighboring pixels, and the background likelihood Q (x ^t )

Calculated by Here, φ (x ^t , z ^t ) is a threshold th that allows a prediction error,

Defined in

In addition, the prediction model, the update is carried out by the observed value y ^t. Here, similar to the probability model update, the pixel value to be used is varied depending on whether the pixel is determined to be the foreground or the background. Details will be described later.

  The final likelihood is determined by integrating the background likelihood calculated with the probability model and the background likelihood calculated with the prediction model. Specifically, by defining an energy function based on a Markov random field and solving the minimization problem, the optimal assignment of binary labels to each pixel is determined. Assuming that the binary label vector is L (l1 to lN), the energy function E (L) is

Can be expressed as

  Here, ν is a set of all pixels, and ν and ε are sets of neighboring pixels. G (li) is a penalty term, H (li, lj) is a smooth term,

Is calculated by

  Here, a graph cut is used to minimize the energy function E (L). Specifically, a node Source (s) corresponding to each pixel of the image and a special node Sink (t) shown in FIG. 4 are created. Here, the node s is the foreground and the node t is the background. The nodes are connected by edges, and the following cost u (i, j) is given to each edge.

  For the pixels that the image processing unit 4 determines to be foreground pixels in the above-described processing, when the probability model and the prediction model are updated with the observed pixel values, when the object is stationary, the pixel values based on the object are gradually used as the background. learn. The image processing unit 1 does not reduce the learning rate of such pixels, but prevents learning at all, thereby preventing a stationary object from blending into the background.

  Also, changes in the surrounding environment, such as brightness, that occurred while the object was stationary, learned and updated changes in the probabilistic model and prediction model for pixels around the stationary object. If the probability model and the prediction model are not updated, there is a high possibility that the background is erroneously detected as an object when the stationary object moves. Therefore, the image processing unit 4 updates the probability model and the prediction model of the pixel determined as the foreground pixel by using the information of the pixel determined as the background pixel.

Specifically, F a set of pixels determined as a foreground pixel, a set of pixels determined as the background and is B, the pixel value yi ^t of the pixel i at time t used to update the probabilistic model,

  Is calculated by Here, c belongs to the set B, and among the pixels determined as the background pixels,

Search for pixels that satisfy.

Θ is a parameter set composed of a probability model of each pixel and a prediction model, where Θ = (μ1 ^t , m ^t , r ^t )
It was.

These are the average value μ1 ^t of the distribution representing the distribution of the pixel values observed so far, the moving average value of the prediction model, and the estimated value of the trend in the probability distribution. For the pixel i determined as the foreground pixel, the image processing unit 4 determines, for each of the probability model and the prediction model of the pixel i, a pixel determined as a background pixel having a probability model similar to them and a prediction model. Explore. Then, by using the searched pixel, the model of the background area shielded by the object is updated. Whether the models are similar may be evaluated by f (Θi, Θj).

  As described above, in the update process of the probabilistic model and the prediction model, for the pixel determined to be the foreground pixel, the pixel value of the pixel is not directly used, but is similar among the pixels determined to be the background pixel. The pixel value of the pixel having the probability model and the prediction model is updated. For example, if the frame image input to the image input unit 3 is the background image shown in FIG. 5A, the image processing unit 4 uses a probability model, And calculate and update the prediction model. On the other hand, if the frame image input to the image input unit 3 is an image capturing the object (vehicle) shown in FIG. 5B, the image processing unit 4 is positioned in the region where the background is captured. For a pixel to be processed, a probability model and a prediction model are calculated and updated based on the pixel value of the pixel. On the other hand, for the pixels located in the area where the vehicle is imaged, the image processing unit 4 is a pixel located in the background area, and the pixel value of the pixel similar to the pixel, the probability model, and the prediction model Is used to update the probability model and the prediction model.

  Therefore, it is possible to appropriately update models (probability models and prediction models) according to environmental changes such as changes in brightness in the imaging area while suppressing the influence of the imaged object on the input frame image. . Thereby, the detection accuracy of the object imaged in the frame image can be improved.

DESCRIPTION OF SYMBOLS 1 ... Image processing unit 2 ... Control part 3 ... Image input part 4 ... Image processing part 4a ... Memory 5 ... Input / output part 10 ... Video camera

Claims (6)

An image input unit for inputting a frame image of an imaging area captured by the imaging device;
For each pixel of the frame image, a probability model storage unit that stores a probability model that models the frequency of the pixel value of the pixel in the past;
For each pixel of the frame image, a prediction model storage unit that stores a prediction model that models the time-series transition of the pixel value of the pixel;
For each pixel of the frame image input to the image input unit, based on the probability model of the corresponding pixel stored in the probability model storage unit, a background pixel whose background is imaged or an object that is not a background A binarized image generation unit that determines which of the foreground pixels being imaged and generates a binarized image to which the determination result is assigned;
An object detection unit for detecting an object located in the imaging area from the binarized image generated by the binarized image generation unit;
For the pixel determined by the binarized image generation unit as a background pixel, the pixel value of the pixel is used to update the probability model and the prediction model for the pixel, and the binarized image generation unit For a pixel that is determined to be a foreground pixel, the pixel value of the pixel that is similar to the probability model and the prediction model and that the binarized image generation unit determines to be a background pixel is used. An image processing unit comprising: a probability model related to the pixel; and a model update unit that updates a prediction model.   The binarized image generation unit stores the probability model of the corresponding pixel stored in the probability model storage unit and the prediction model storage unit for each pixel of the frame image input to the image input unit. The image processing unit according to claim 1, wherein the image processing unit determines whether the pixel is a background pixel or a foreground pixel based on a prediction model of the corresponding pixel.   The binarized image generation unit is a background pixel based on a probability model of a corresponding pixel stored in the probability model storage unit for each pixel of the frame image input to the image input unit. The second likelihood of a background pixel is calculated based on the prediction model of the corresponding pixel stored in the prediction model storage unit, and the calculated first likelihood, and The image processing unit according to claim 2, wherein the second likelihood is integrated to determine whether the pixel is a background pixel or a foreground pixel.   The image processing unit according to claim 1, wherein the probability model is a density function having a Gaussian distribution.   The image processing unit according to claim 1, wherein the prediction model is a model based on an exponential smoothing method. Probability that the probability model storage unit models the frequency of the pixel value of the pixel in the past for each pixel of the frame image with respect to the frame image of the imaging area captured by the imaging device that is input to the image input unit Storing the model, and the prediction model storage unit stores, for each pixel of the frame image, a prediction model obtained by modeling a time-series transition of the pixel value of the pixel,
For each pixel of the frame image input to the image input unit, based on the probability model of the corresponding pixel stored in the probability model storage unit, a background pixel whose background is imaged or an object that is not a background A binarized image generation step of determining which of the foreground pixels being imaged and generating a binarized image to which the determination result is assigned;
An object detection step of detecting an object located in the imaging area from the binarized image generated in the binarized image generation step;
For a pixel determined to be a background pixel in the binarized image generation step, the probability model and prediction model for the pixel are updated using the pixel value of the pixel, and in the binarized image generation step For the pixel determined to be a foreground pixel, the pixel value of the pixel that is similar to the probability model and the prediction model and determined to be a background pixel in the binarized image generation step is used. An image processing program for causing a computer to execute a probability update model for the pixel and a model update step for updating a prediction model.

Images