JP2006098119A - Object detector, object detection method, and object detection program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object detector capable of stably detecting an object moving in a designated direction even when the number of the objects is large. <P>SOLUTION: A frame image acquisition part 1 acquires a plurality of frame images from picture data. An optical flow calculation part 4 calculates the optical flow of each pixel in the frame images. A background image preparation part 2 generates a background image in which the moving object is removed from the plurality of frame images. An image difference calculation part 3 calculates the difference between the background image and the frame image to generate a background differential image. A peak detection part 6 adds a pixel value of the background differential image for each axis of the designated direction to detect the maximum value. A direction determination part 7 estimates the moving direction of the maximum value. A peak image generation part 8 generates a peak image in which the maximum value on the axis of the designated direction and the moving direction data acquired from the plurality of frame images are recorded in time series. A track determination part 9 detects the track formed by the maximum value having the designated direction in the peak image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an object detection apparatus, an object detection method, and an object detection program for detecting an object such as a person moving in a specified direction in a large-scale facility such as an airport or a station.

  Conventionally, in large-scale facilities such as airports and stations, there is a need to detect an object such as a person moving in a specified direction from the viewpoint of safety management and crime prevention.

  Systems that count area sensors (cameras), line sensors, etc. directly downward, and count people's movement direction and number of people using signal processing and image processing technologies have been put into practical use. There was a problem that it was necessary to install it and the initial investment was large.

  On the other hand, a large number of surveillance cameras are already installed at stations, airports, and the like, and if an object such as a person moving in a designated direction can be detected by using these cameras, an inexpensive system can be constructed. For this reason, expectation is given to the retrograde person detection technique using the existing surveillance camera.

As a method for detecting an object such as a person moving in a specified direction, it is conceivable to use processing (tracking) for tracking an object in an image (for example, Patent Document 1, Patent Document 2, and Patent Document 3). , Patent Document 4 and Patent Document 5). Note that tracking is a process for obtaining a trajectory of an object (a trajectory of (x, y, t)) when the pixel position of the image is (x, y) and the time is t.
Japanese Patent Laid-Open No. 11-083530 Japanese Patent Laid-Open No. 10-105712 Japanese Patent Laid-Open No. 07-160856 Japanese Patent Laid-Open No. 06-266840 JP 07-49952 A

  However, in the above-described conventional technology, when the surveillance camera is installed obliquely downward, in such a camera image, there is a problem that an object is superimposed (hidden) on the image and tracking becomes extremely unstable. . At present, a camera that operates stably even when it is installed obliquely downward and there are many objects has not been realized.

  The present invention has been made in view of such circumstances, and its purpose is to use an existing surveillance camera installed obliquely downward to detect an object that moves stably in a specified direction even when there are many objects. An object of the present invention is to provide an object detection device, an object detection method, and an object detection program capable of detection.

  In order to solve the above-described problem, the present invention provides an object detection device that detects an object moving in a specified direction by acquiring a frame image from video data and performing image processing on the frame image. Frame image acquisition means for acquiring a plurality of frame images from data, optical flow calculation means for calculating the optical flow of each pixel in the frame image, and a background image from which moving objects are excluded from the plurality of frame images A background image creating unit; an image difference calculating unit that calculates a difference between the background image and the frame image to generate a background difference image; and a pixel value of the background difference image is added for each axis in the specified direction. A peak detecting means for detecting a local maximum value, and a value of a component in the designated axis direction of the optical flow as an axis in the designated direction. And a direction determination means for estimating the moving direction of the maximum value, and a peak image obtained by recording the maximum value and moving direction data on the specified direction axis in time series, obtained from the plurality of frame images. An object detection apparatus comprising: a peak image generation unit to generate; and a trajectory determination unit that detects a trajectory formed by a maximum value having the specified direction in the peak image.

  The present invention is characterized in that, in the above-mentioned invention, alarm generation means for issuing an alarm when an object moving in the designated direction is detected by the trajectory determination means.

  According to the present invention, in the above invention, the trajectory determination unit uses a Hough transform that detects a straight line from a fragmentary point sequence when determining the trajectory of the object based on the specified direction in a predetermined time range. It is characterized by doing.

  The present invention obtains a plurality of frame images from the video data in an object detection method for detecting an object moving in a specified direction by acquiring a frame image from the video data and performing image processing on the frame image. Calculating an optical flow of each pixel in the frame image, generating a background image by removing moving objects from the plurality of frame images, calculating a difference between the background image and the frame image, and calculating a background difference image Generating, adding a pixel value of the background difference image for each axis in the designated direction to detect a local maximum value, adding a value of the component in the designated axis direction of the optical flow for each axis in the designated direction, The movement direction of the local maximum value is estimated, and the local maximum value and the movement direction data on the specified direction axis acquired from the plurality of frame images are recorded in time series. Generating an image, an object detection method characterized by detecting a trajectory the maximum value with the given direction is formed in the peak image.

  The present invention acquires a plurality of frame images from the video data to a computer of an object detection device that detects a moving object in a specified direction by acquiring a frame image from the video data and performing image processing on the frame image. Obtaining a difference between the background image and the frame image; obtaining an optical flow of each pixel in the frame image; generating a background image by removing moving objects from the plurality of frame images; Calculating a background difference image; adding a pixel value of the background difference image for each axis in the specified direction to detect a maximum value; and a value of a component in the specified axis direction of the optical flow For each axis in the designated direction, estimating the moving direction of the maximum value, and the plurality of frame images Generating a peak image obtained by recording time-series local maximum values and movement direction data on the designated direction axis, and detecting a locus formed by the local maximum value having the designated direction in the peak image. And an object detection program characterized in that

  According to the present invention, the object detection device acquires a plurality of frame images from video data, calculates an optical flow of each pixel in the frame image, and generates a background image by excluding moving objects from the plurality of frame images. . Next, a difference between the background image and the frame image is calculated to generate a background difference image, and the maximum value is detected by adding the pixel values of the background difference image for each axis in the designated direction. In addition, the value of the component in the specified axis direction of the optical flow is added for each axis in the specified direction, the moving direction of the maximum value is estimated, and the maximum value and moving direction data on the specified direction axis acquired from a plurality of frame images Is generated in time series. And it is the structure which detects the locus | trajectory which the maximum value with a designated direction forms in a peak image. Therefore, even if the video data is captured by an existing surveillance camera installed obliquely downward, an object moving in a designated direction can be detected by using the optical flow.

  Further, according to the present invention, the object detection device is configured to issue an alarm when it is determined that there is an object moving in the specified direction based on the trajectory. Therefore, it is possible to know the presence of an object that moves in the direction specified by the alarm.

  According to the present invention, the object detection device is configured to use the Hough transform for detecting a straight line from a fragmentary point sequence when estimating the locus of the object based on the specified direction. Therefore, it is possible to extract a straight line indicating the direction from the movement trajectory configured by the point sequence, and it is possible to easily detect an object moving in the designated direction from the difference in the inclination of the straight line.

  Hereinafter, the best mode for carrying out the present invention will be described.

A. Embodiments The present invention is realized by acquiring a frame image that is an image for each moment from video data such as a video signal or a video file, and performing image processing on the acquired frame image. Accordingly, various apparatuses such as a PC having an image input function and a dedicated image processing board or an image processing apparatus incorporating an image processing procedure can be applied as an apparatus for carrying out the present invention. A specific apparatus is not limited.

  The present invention detects that a person, an automobile, or some other object has moved in a predetermined direction. As an example, a case is assumed where a person who moves in the opposite direction in a place where the moving direction is limited, such as a passage in an airport, is detected using a monitoring camera image. In the following, in order to facilitate understanding, the explanation will be made taking the case of retrograde detection as an example.

  In the present invention, it is assumed that the movement of the person to be detected is observed as a movement in the up / down / left / right direction (which may be oblique) on the image. Therefore, when a person moves in the real space in the direction of the optical axis of the camera (that is, in the depth direction of the image), although the size on the image changes, the position remains unchanged and no movement is observed. Cases are not covered by the present invention.

The basic procedure and concept of the present invention are shown below. Here, a case where the passage is photographed from the side, that is, a case where the moving direction is the left and right (x-axis direction) of the image will be described as an example. A rough processing procedure is shown in 1) to 4) below, and a basic concept is shown thereafter.
1) Estimate the candidate position of the object on the x-axis for each moment.
2) The movement direction in the x-axis direction of the object at the candidate position is estimated.
3) In the image obtained by accumulating the candidate position and the estimated moving direction for each moment in time series, the locus (xt locus) of the candidate position in the direction to be detected is estimated.
4) For the estimated xt trajectory, the probability of the trajectory is evaluated, and if it is determined to be probable (detected as a retrograde).

  In 1) and 2), it is possible to stably estimate the candidate position and the moving direction of the person by roughly grasping the moving direction as one dimension. For example, in the case where a plurality of people cross and overlap each other on the image, it is difficult to stably obtain the two-dimensional position (x, y) on each person's image. The value of x can be estimated more stably by using the above information in an additive manner (projection of background difference, which will be described later). However, the estimation results of each instantaneous candidate position and moving direction in 1) and 2) are not always correct, and may not be detected at a certain moment or erroneously detected at a certain moment. This problem is solved by applying an algorithm that can detect a trajectory from a fragmentary point sequence in 3). Details are shown in A-1 and A-2.

  With such a procedure, it is possible to stably detect an object moving in a specified direction, such as detection of a retrograde person in a crowd, using an existing surveillance camera installed obliquely downward.

A-1. Configuration of Embodiment FIG. 1 is a block diagram showing a configuration of a retrograde person detection device which is an embodiment of an object detection device according to the present invention. In FIG. 1, a frame image acquisition unit 1 acquires a frame image from video data. The background image creation unit 2 calculates a background image from a plurality of frame images. Here, the background image is an image in which no moving object (in this example, a person) exists. The image difference calculation unit 3 calculates a pixel value difference (or an absolute value of the difference) between the nth (current time) frame image and the background image. By taking the difference (absolute value) between the pixel values of the background image and the frame image at the current time, the difference value is 0 or a difference value near 0 at the current time, and the large difference value is at the place where the person is present. A background difference image which is an image as given is calculated.

  Also, the optical flow calculation unit 4 calculates an optical flow (motion vector) from the frame image. The optical flow is calculated as a velocity vector (u (x, y), v (x, y)) where the coordinates on the image are (x, y). Here, u (x, y) is the velocity component in the x direction observed at the pixel (x, y), and v (x, y) is the y direction observed at the pixel (x, y). It is a velocity component. When attention is paid only to the movement in the left-right direction as in this example, the y component v (x, y) of the velocity vector is unnecessary, and only the x component u (x, y) is used. The optical flow includes a rightward flow image in which a pixel value u (x, y) is given to a pixel in which the x component u (x, y) of the optical flow has a positive value, and the pixel values of other pixels are 0. Two pixels of a left-facing flow image in which the pixel value −u (x, y) is given to a pixel having a negative value in the x component u (x, y) of the optical flow and the pixel values of the other pixels are 0 I express it as an image.

  In this way, the two images are represented by aligning the representation of the images so that they can be performed by a common process when the projection of the background image and the projection of the optical flow (x component) are obtained in the image projection unit described later. Therefore, creating two images is not an essential configuration in the present invention.

  The image projection unit 5 has a function of calculating projection data, which is an addition value for each x coordinate, by adding the pixel value of an arbitrary image to the x-axis. By adding for each coordinate, it is output as background difference projection data. FIG. 6 shows background difference projection data created by projecting the number of pixels for each x coordinate (vertical direction) onto the x axis in this way. Further, the image projection unit 5 adds the right flow image and the left flow image for each x coordinate (vertical direction), and outputs the result as flow projection data. The peak detection unit 6 detects all positions of the maximum value (peak) of the projection value from the background difference projection data.

The direction determination unit 7 determines the direction (left or right) at each maximum value (peak) position according to the maximum value (peak) position detected by the peak detection unit 6 and the flow projection data. The peak image generation unit 8 generates a peak image in which the peak positions are plotted according to the direction of the maximum value (peak) position. The trajectory determination unit 9 detects a peak position sequence (plural) that draws a straight line from the peak position plotted in the peak image by a point sequence detection algorithm such as a Hough transform, and for each line candidate, Calculate the peak score. The alarm generation unit 10 determines whether there is a person going backward according to the number of peak points and issues an alarm.
Note that peak detection and estimation of the moving direction can be made more stable by applying a smoothing filter such as a one-dimensional Gaussian filter to the background difference projection data and the flow projection data.

  The above-described frame image acquisition unit corresponds to the frame image acquisition unit 1, the optical flow calculation unit corresponds to the optical flow calculation unit 4, the background image generation unit corresponds to the background image generation unit 2, and the image difference The calculation means corresponds to the image difference calculation unit 3. Further, the peak detection means corresponds to the image projection unit 5 and the peak detection unit 6, the direction determination unit corresponds to the direction determination unit 7, and the peak image generation unit corresponds to the peak image generation unit 8, and the locus determination. The means corresponds to the locus determination unit 9.

A-2. Operation of Embodiment Next, the operation of the above-described embodiment will be described. Here, FIG. 2 is a flowchart for explaining the operation of the retrograde detection device according to the present embodiment. First, the frame image acquisition unit 1 acquires a plurality of frame images from the video data. Next, the background image creation unit 2 creates a background image from the plurality of frame images (S1). FIG. 4 is a diagram of the background image created in S1. Various methods for creating a background image have been proposed. As an example, there is a method of detecting a mode value (mode) of a pixel value for each pixel using a plurality of frame images.

  Next, from the background image and the n-th frame image, the image difference calculation unit 3 gives 0 or a difference value close to 0 at the current time and a large difference value at the position where the person exists. A background difference image that is an image to be obtained is calculated. In calculating the background difference image, the difference between the nth frame image and the background image or the absolute value of the difference is calculated, and then smoothed by a smoothing filter such as a Gaussian filter. You may perform the process which values. By performing such processing, it is possible to obtain a background difference image showing a place where a person exists as a stable lump. (S2) FIG. 5 is a diagram of the background difference image created in S2.

  Next, background difference projection data is generated by the image projection unit 5 by adding the pixel values of the background difference image for each x coordinate (S3). FIG. 6 is a diagram of the background difference projection data created in S3. Next, the peak detection unit 6 detects the position of the maximum value (peak) of the projection value from the background difference projection data (S4). The peak positions are x1, x2, x3,..., Xj,.

  FIG. 3 is an example of the (n−1) th frame image and the nth frame image among the video data composed of a plurality of frame image sequences. The optical flow calculation unit 4 calculates an optical flow (motion vector) from the (n-1) th frame image and the nth (current time) frame image (S5). Further, the image projection unit 5 creates an optical flow image in the left-right direction from the optical flow (S6). FIG. 7 is an example of each flow image. (A) is a flow image in the left direction, and (b) is a flow image in the right direction. A flow in which the velocity vector in the x direction, u (x, y) is greater than 0, is a flow image in the right direction, and a flow in which u (x, y) is less than 0 is a flow image in the left direction. Further, the image projecting unit 5 adds the right flow image and the left flow image for each x coordinate (vertical direction), and outputs the result as left and right flow projection data (S7). Here, the left flow projection data is L (x), and the right flow projection data is R (x). FIG. 8 shows an example of the flow projection data, where (a) is the left flow projection data and (b) is the right flow projection data.

  Next, the direction determination unit 7 determines the direction (left or right) at each peak position according to the peak position from the peak detection unit 6 and the flow projection data (S8). The determination method is a leftward peak if L (x)> R (x), and a rightward peak if L (x) <R (x). Next, the peak image generation unit 8 changes the pixel value at the peak position depending on the estimated direction (for example, the pixel value is 1 for the peak in the right direction, the pixel value is 2 for the peak in the left direction, 0 for the non-peak pixel, etc. ) Is generated (S9). FIG. 9 is an example of the generated peak image. The peak image is obtained by accumulating the peak position and direction acquired from the past frame in time series. In the example of the peak image in FIG. 9, the nth is the peak position acquired from the frame at the current time. Yes, n-1 below is a peak position acquired from a past frame.

Next, the trajectory determination unit 9 sets a line candidate where a retrograde direction (in this case, a peak to which a pixel value indicating the right direction is given) is aligned on a straight line from the peak position plotted in the peak image. It detects (for example, straight line L in FIG. 9) (S10). Hough transform can be used as one of means for realizing this straight line candidate. The Hough transform is an image processing algorithm for detecting straight line candidates from a fragmentary point sequence. Next, the number of rightward peak points existing in the vicinity for each obtained straight line candidate is counted (S11). If the number of peak points is greater than a preset threshold value, it is determined that there is a high possibility that the trajectory is caused by a person's retrograde movement, and an alarm is issued (S12). Thereafter, the process proceeds to the next frame (S13), and the above-described processing is repeated.
In addition, when detecting a straight line candidate, the inclination of the straight line to be detected can be limited to some extent. Taking FIG. 9 as an example, since a person (object) moving to the right should have an upward slope in the peak image, it is only necessary not to detect a downward straight line. Further, when the moving speed that is realistically assumed is limited to a certain range, only a straight line having an inclination corresponding to the range may be detected. By limiting the straight line candidates in this way, erroneous detection can be suppressed.

  In the above-described embodiment, for the sake of simplification of explanation, the case of detecting the retrograde person, particularly the case where the human movement is observed as the left-right movement is described as an example. Even if the movement direction of the person is not a perfect left-right direction, for example, even if the correct movement direction is toward the lower left and the reverse direction is toward the upper right, the process is performed by projecting to the x axis in the same manner as described above. By doing so, it is possible to detect retrograde. On the other hand, when the movement on the image is only in the vertical direction and the movement in the horizontal direction is not meaningful, such as when photographing the passage straight in the back, the same can be done by switching the x axis and the y axis described above. Applicable to. Further, it is obvious that the present invention can be generalized to “movement in a specified direction” without being limited to “retrograde”.

  Moreover, although examples of images in the respective processes are shown in FIGS. 4, 5, and 7, it goes without saying that displaying these images is not an essential configuration in the present invention.

  Further, the background projection data and the left and right flow projection data are created in the image projection unit 5 and the direction of the maximum value and the maximum value is detected. However, the image projection unit 5 is not necessarily limited to creating these three pieces of image data. It is clear that the projection image data creation process can be omitted if the maximum value and the direction at the maximum value can be finally detected.

  Note that one of the conditions under which the present invention operates best is when the moving direction of interest on the image is left and right (up and down), and the target object is vertically long (landscape). For example, it is a case where a passage is photographed from substantially the side and a movement of a person (which appears vertically long on the image) in the left-right direction is targeted. In such a case, candidate position (peak) detection by projection can be performed with high accuracy, so that more stable detection can be expected. Of course, even if this condition is not satisfied, the present invention can be applied and is not limited to this.

  In the above-described embodiment, the frame image acquisition unit 1, the background image creation unit 2, the image difference calculation unit 3, the optical flow calculation unit 4, the image projection unit 5, the peak detection unit 6, the direction determination unit 7, and the peak image The generation unit 8 and the trajectory determination unit 9 are executed in a computer system. The frame image acquisition unit 1, the background image creation unit 2, the image difference calculation unit 3, the optical flow calculation unit 4, the image projection unit 5, the peak detection unit 6, the direction determination unit 7, the peak image generation unit 8, the locus described above. A series of processing steps by the determination unit 9 is stored in a computer-readable recording medium in the form of a program, and the above-described processing is performed by the computer reading and executing this program. That is, the frame image acquisition unit 1, the background image creation unit 2, the image difference calculation unit 3, the optical flow calculation unit 4, the image projection unit 5, the peak detection unit 6, the direction determination unit 7, the peak image generation unit 8, and the locus determination unit 9 is realized when a central processing unit such as a CPU reads the above program into a main storage device such as a ROM or a RAM, and executes information processing / arithmetic processing.

  Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

It is a block diagram which shows the structure of the retrograde person detection apparatus of this embodiment. It is a flowchart for demonstrating operation | movement of the retrograde person detection apparatus by the embodiment. It is the figure which showed the video data which consists of a some frame image sequence by the embodiment. It is the figure which showed the background image by the embodiment. It is the figure which showed the background difference image by the embodiment. It is the figure which showed the background difference projection data by the same embodiment. It is the figure which showed the flow image by the embodiment. It is the figure which showed the flow projection data by the embodiment. It is the figure which showed the peak image by the same embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Frame image acquisition part 2 ... Background image creation part 3 ... Image difference calculation part 4 ... Optical flow calculation part 5 ... Image projection part 6 ... Peak detection part 7 ... Direction determination part 8 ... Peak image generation part 9 ... Trajectory determination part 10 ... Alarm generator

Claims (5)

In an object detection device that detects a moving object in a specified direction by acquiring a frame image from video data and performing image processing on the frame image,
Frame image acquisition means for acquiring a plurality of frame images from the video data;
An optical flow calculating means for calculating an optical flow of each pixel in the frame image;
Background image creating means for generating a background image in which moving objects are excluded from the plurality of frame images;
Image difference calculating means for calculating a difference between the background image and the frame image to generate a background difference image;
Peak detection means for detecting the maximum value by adding the pixel values of the background difference image for each axis in the designated direction;
Direction determining means for adding the value of the component in the designated direction of the optical flow for each axis of the designated direction, and estimating the moving direction of the maximum value;
Peak image generation means for generating a peak image obtained by recording time-series local maximum values and movement direction data on the specified direction axis obtained from the plurality of frame images;
Trajectory determination means for detecting a trajectory formed by the maximum value having the specified direction in the peak image;
An object detection device comprising:   The object detection apparatus according to claim 1, further comprising: an alarm generation unit that issues an alarm when an object moving in the designated direction is detected by the trajectory determination unit.   The trajectory determination means uses a Hough transform that detects a straight line from a fragmentary point sequence when determining the movement trajectory of an object based on the specified direction in a predetermined time range. Or the object detection apparatus of 2. In an object detection method for detecting an object moving in a specified direction by acquiring a frame image from video data and performing image processing on the frame image,
Obtaining a plurality of frame images from the video data, calculating an optical flow of each pixel in the frame image, generating a background image by excluding moving objects from the plurality of frame images, the background image and the frame image To calculate a difference between the pixel and the pixel value of the background difference image is added for each axis in the specified direction to detect a maximum value, and the value of the component in the specified axis direction of the optical flow For each axis in the specified direction, estimating the moving direction of the maximum value, and a peak image in which the maximum value and moving direction data on the specified direction axis acquired from the plurality of frame images are recorded in time series. And detecting a locus formed by a maximum value having the specified direction in the peak image. By acquiring a frame image from video data and performing image processing on the frame image, the computer of the object detection device that detects an object moving in a specified direction,
Obtaining a plurality of frame images from the video data;
Calculating an optical flow of each pixel in the frame image;
Generating a background image by removing moving objects from the plurality of frame images; generating a background difference image by calculating a difference between the background image and the frame image;
Adding a pixel value of the background difference image for each axis in the designated direction to detect a maximum value;
Adding a value of the component in the designated axis direction of the optical flow for each axis in the designated direction, and estimating a moving direction of the maximum value;
Generating a peak image in which time-series recorded local maximum values and movement direction data on the specified direction axis obtained from the plurality of frame images;
Detecting a locus formed by a maximum value having the specified direction in the peak image;
An object detection program characterized by causing

Images