JP6607630B2 - Moving object extraction apparatus, method and program - Google Patents

Description

  The present invention relates to a moving object extraction apparatus, method, and program, and in particular, even when a plurality of moving objects exist in a moving image captured by a camera and an overlap occurs between the moving objects, each moving object is individually detected. The present invention relates to a moving object extraction apparatus, method, and program that can be accurately extracted.

  Monitoring and tracking of a moving object using a moving image photographed by a camera and identifying the movement of the moving object are performed in various fields. For this purpose, it is desirable to accurately extract a moving object even from moving images in various situations.

  The methods for extracting a moving object from a moving image are roughly divided into the following three methods.

(1) Method based on background difference A method for separating a moving object as a foreground and a background by obtaining a difference between frames. For example, Patent Document 1 describes an object extraction technique based on background image estimation. .

(2) Method based on motion vector classification Using motion vector detection technology, calculate the direction and speed of moving objects from moving images, analyze the information, classify the motion vectors for each moving object, This is a method of extracting each moving object. For example, Patent Literature 2 describes this technique.

(3) Method based on motion boundary detection This is a method for detecting the motion boundary for each moving object and estimating the area of each moving object as a foreground based on motion vector detection technology. Patent Document 1 describes this technique.

JP 2011-13978 A JP 2010-211578 A

Papazoglou, A., & Ferrari, V. Fast object segmentation in unconstrained video.IEEE International Conference on Computer Vision (ICCV), pp. 1777-1784, 2013.

  In the prior art described in Patent Document 1-2 and Non-Patent Document 1, it is difficult to accurately extract a moving object from a moving image captured by a camera in various situations.

  For example, the method based on the background difference described in Patent Document 1 has a problem that it can be applied only when the camera is fixed or when the movement is slow even if it moves. In addition, when there is an overlap between a plurality of moving objects, there is a problem that it is difficult to extract each moving object separately.

  In the method based on the classification of motion vectors described in Patent Document 2, the motion vector for the same moving object is required to be uniform in all parts. There is a problem that it is difficult to extract a moving object with different motion vectors and deformed shapes. In addition, when there is an overlap between a plurality of moving objects, there is a problem that it is difficult to extract each moving object separately.

  The method based on the boundary detection of motion described in Non-Patent Document 1 can be applied both when the camera moves and when the motion of the moving object is partially different, but there is an overlap between a plurality of moving objects. When this occurs, there is a problem that it is difficult to extract each moving object separately.

  The object of the present invention is to solve the above-mentioned problem, and even when the movement of each part of the moving object is different or when there is an overlap between a plurality of moving objects, the moving object extraction that can accurately extract each moving object separately To provide an apparatus, a method, and a program.

To solve the above problems, the present invention is a mobile object extraction unit for extracting a moving object from the moving image, the moving object detection unit for detecting a moving object region in a moving picture, the moving object region and the moving object A moving object information calculation unit that calculates information about each part that moves differently in the region as moving object region information, and information on the moving object region calculated by the moving object information calculation unit for all registered moving objects It is determined whether or not the moving object region corresponds to a plurality of moving objects, and if it is determined that they do not correspond, the moving object region information is used as information about the moving object. update the information that is already registered for the new registration or the moving object relative to the object, when it is determined the corresponding a, the moving object information calculation hand The mobile information about each part calculated from separated into the plurality of moving objects registered information and area information of each moving object a moving object area information by comparing for each site with respect to the A moving body information management unit that updates information already registered for an object is provided.

  Here, the moving object detection unit detects a motion vector and a feature point group in the moving vector from at least two frame images, classifies the feature point group according to the detected motion vector, and classifies the feature points A non-convex shape is extracted from the group to generate the contour of the moving object region, the approximate region of the moving object region is estimated from the generated contour, the approximate region of the estimated moving object region is used as the foreground, and the image It is preferable to extract the moving object region by dividing the whole into a foreground and a background.

  Further, the moving object information calculation unit calculates the geometric center of the moving object region as the position of the moving object region, calculates the number of pixels of the moving object region as the size of the moving object region, and moves each part of the moving object region. Each part is classified according to the speed and direction of the vector, the geometric center of each classified part is calculated as the position of each part, and the moving speed and moving direction of each part are calculated from the motion vector of each classified part. It is also preferable.

  In addition, it is preferable that the moving body information management unit assigns an ID to each moving object and manages information about each moving object.

  Further, the moving body information management unit determines whether or not the moving object area information corresponds to a plurality of moving objects based on a cover relationship between the moving object area and the area of each registered moving object. Is also preferable.

  Note that the present invention can be realized not only as a moving object extraction apparatus, but also as a moving object extraction method that sequentially executes processing in each unit, or as a program that causes a computer to function as each unit of the moving object extraction apparatus.

  According to the present invention, each moving object can be accurately and separately extracted even in various situations such as when the movement of each part of the moving object is different or when there is an overlap between a plurality of moving objects.

It is a block diagram which shows one Embodiment of the mobile body extraction apparatus which concerns on this invention. It is explanatory drawing which shows the example of the time-sequential moving image image | photographed with the camera. It is explanatory drawing which shows the process in a moving object detection part. It is explanatory drawing which shows an example of the result of having classified the motion vector in each site | part of a moving object. It is a flowchart which shows the process in a moving object ID and an information management part. It is explanatory drawing of the process in a moving object ID and an information management part. It is explanatory drawing of isolation | separation of a shielding moving object.

  The present invention will be described below with reference to the drawings.

  The present invention extracts a moving object from a time-series moving image taken by a camera. In the following description, it is assumed that the moving object is a person.

  FIG. 1 is a block diagram showing an embodiment of a moving object extraction apparatus according to the present invention.

  The moving object extraction device 10 includes a moving object detection unit 11, a moving object information calculation unit 12, a moving object ID and an information management unit 13, and a person as a moving object from a time-series moving image captured by one camera. To manage the ID and information of each moving object.

  The moving object ID and information management unit 13 includes a moving object update unit 15 and a moving object registration unit 16, and the moving object update unit 15 includes a shielding moving object separation unit 14.

  Note that each unit of the moving object extraction apparatus 10 can be realized by software using one or a plurality of processors, and can also be realized by hardware.

  The moving object detection unit 11 uses, for example, a motion vector detection method, a clustering algorithm based on point cloud density, a non-convex shape extraction method, and an algorithm that divides the entire image into a foreground and a background, To detect. The specific processing here will be described in detail later, but the moving object region can be accurately detected by using the above technique together. When an overlap (hereinafter referred to as “shielding”) occurs between a plurality of moving objects (persons), an area including them is extracted as one moving object area.

  The moving object information calculation unit 12 calculates information about the moving object region detected by the moving object detection unit 11 and each part that moves differently in the moving object region.

  The moving object ID and information management unit 13 compares the information calculated by the moving object information calculation unit 12 with the already registered information, and determines whether the detected moving object is unregistered and newly registers the area information. Update the registered information assuming that the detected moving object is already registered, or register the detected moving object after separating it into each moving object as including a plurality of moving objects that are occluded Determine whether to update the information.

  Registered means that the same moving object is already detected in a frame before the current frame, and the information of the moving object is already stored in the moving object ID and information management unit 13 or in an external database (not shown). This means that it is registered, and includes not only new registrations but also updates.

  Here, when it is determined that the detected moving object is unregistered, the moving object is set as a new moving object, and the moving object registration unit 16 assigns a new ID thereto and newly registers the information.

  Further, when it is determined that the detected moving object is already registered, the previous ID is taken over and the already registered information is updated with the newly calculated information.

  In addition, when it is determined that the detected moving object includes a plurality of moving objects in which shielding is occurring, the shielding moving object separation unit 14 separates the information into information on individual moving objects and separates them. Based on the information, after calculating the necessary information, the corresponding registered information is updated. In this case, the mobile unit ID takes over the previous ID. As will be described later, since a plurality of moving objects with occlusion are detected after an individual moving object is detected before that, the information is updated here. ing.

  Hereinafter, the processing in each part of FIG. 1 will be described in detail.

  FIG. 2 is an example of a time-series moving image captured by a camera, and shows a time-series moving image when two moving persons a and b are captured by a camera. Here, time t1 is an arbitrary time in which two persons a and b exist in the screen and no shielding occurs between the persons a and b, and a time t2 includes shielding between the persons a and b. It is the time immediately before. In these images, persons a and b are extracted separately. The time t2 + 1 is the moment when the shielding occurs, and the persons a and b are extracted as one area in this image. In a time-series moving image, first, a person a appears, and then another person b appears, approaches the person a, and a shielding occurs between them. Yes, here, such time-series moving images are targeted.

  The moving object detection unit 11 first detects a motion vector of a feature point (point group) from at least two frame images in a time-series moving image. Here, the two frame images are two consecutive images selected from the time-series moving images. The feature point is a pixel on the frame image where the motion vector is detected. This motion vector is described in, for example, “Farneback, Gunnar.“ Two-frame motion estimation based on polynomial expansion. ”In Image Analysis. Springer Berlin Heidelberg, pp. 363-370, 2003.” (Reference 1). The optical flow method based on the luminance value of each pixel of the image and the neighboring area of each pixel, and "Sundaram, Narayanan, Thomas Brox, and Kurt Keutzer." Dense point trajectories by GPU-accelerated large displacement optical flow. "Computer Vision-ECCV 2010. Springer Berlin Heidelberg, pp. 438-451, 2010. ”(Reference 2) can be detected using the motion vector detection method based on the high-density point cloud, but it can be detected by other methods. May be.

  FIG. 3A shows a moving point group detected from the image at time t1 in FIG. Here, a point group Pa in the area of the person a, a point group Pb in the area of the person b, and point groups BG1, BG2, and BG3 in the area with a slight background are detected as moving point groups.

  The displacement of each point group can be estimated from the motion vectors of these point groups. A point group having a small displacement can be identified as a point group for the background, and other points can be identified as a point group for a moving object as the foreground.

  Therefore, the moving object detection unit 11 then classifies (clusters) the detected point group. For example, “Ester, Martin, et al.“ A density-based algorithm for discovering clusters in large spatial databases with noise. ”Kdd, vol. 96, no. 34, 1996.” (reference 3) A clustering algorithm based on the described point cloud density can be used, but other methods may be used.

  FIG. 3 (b) shows the result of classifying the point group of FIG. 3 (a). Here, the point group Pa corresponds to the person a, the point group Pb corresponds to the person b, and the point groups BG1, BG2, and BG3 correspond to a slight area of the background. As described above, the point groups BG1, BG2 and BG3 may be detected for a small area of the background in the image photographed by the camera, but the number of points in the point groups BG1, BG2 and BG3 is small. A point cloud is classified by a clustering algorithm based on the group density (Reference 3). Further, for example, when the number of points of the classified point group is equal to or less than a preset threshold value, the point cloud can be identified as a background.

  The moving object detection unit 11 further generates a contour of the moving object region from the detected point group. For example, “F. Bernardini and C. Bajaj.“ Sampling and reconstructing manifolds using alpha-shapes. ”In Technical Report CSD-TR-97-013, Dept. Comput. Sci., Purdue Univ., West This can be realized by extracting a non-convex shape from the detected point group using the method described in “Lafayette, 1997.” (Reference 4), but other methods may be used. FIG. 3C shows a contour extracted from the point group shown in FIG.

  The moving object detection unit 11 further estimates an approximate moving object region from the relationship between the pixels on the frame image and the contour created as described above. For example, “Hormann, Kai, and Alexander Agathos.“ The point in polygon problem for arbitrary polygons. ”Computational Geometry, vol. 20 (3), pp. 131-144, 2001.” (reference 5) By calculating as a point problem on the polygon from the relationship between the pixel on the image and the created contour, an approximate moving object region as shown in FIG. 3 (d) can be estimated, but other methods may be used.

  The moving object detection unit 11 further uses an approximate moving object region estimated as described above as a foreground, and divides the entire image into a foreground and a background, as shown in FIG. 3 (e). Extract regions accurately. For example, "Rother, Carsten, Vladimir Kolmogorov, and Andrew Blake." Grabcut: Interactive foreground extraction using iterated graph cuts. "ACM Transactions on Graphics (TOG), vol. 23 (3), pp. 309-314, 2004. ”(Reference 6) is known and can be used, but other methods may also be used.

  The moving object information calculation unit 12 calculates moving object information including the position and size of the moving object region detected by the moving object detection unit 11, the position of each part with different movement, and the moving speed and moving direction thereof.

  The position of the moving object area can be obtained by extracting the outline of the moving object area detected by the moving object detecting unit 11 and calculating the center point of the polygon in the extracted outline. For example, using the technique described in “Bourke, Paul.“ Calculating the area and centroid of a polygon. ”URL. Paulbourke.net/geometry/polygonmesh/, 1988.” (Reference 7) The center point is calculated as the position of the moving object region. The position of the moving object region may be calculated by other methods.

  The size of the moving object region can be calculated by calculating the number of all pixels in the moving object region detected by the moving object detection unit 11.

  The moving speed v and moving direction d (Δx, Δy) of each part of the moving object region can be estimated from the motion vector detected by the moving object detection unit 11. For example, the feature point pt-1 (xt-1, yt-1) detected in the frame image at time t-1 and the feature point detected in the frame image at time t by the optical flow method described in the above-mentioned document 1 When pt (xt, yt) is a pair of trajectories, the moving speed vt and the moving direction dt (Δxt, Δyt) can be calculated according to the following equations (1), (2), (3).

  The new feature quantity wt (xt , yt, vt, Δxt, Δyt). Then, a new feature amount wt is calculated for all the motion vectors and classified. This classification is described in “Kailing, Karin, Hans-Peter Kriegel, and Peer Kroger.“ Density-connected subspace clustering for high-dimensional data. ”Proc. SDM, vol. 4, 2004. (Reference 8). This can be done by a high-dimensional vector clustering method called SUBCLU. FIG. 4 shows an example of a result obtained by classifying motion vectors in each part of a moving object.

  By averaging the new feature quantities classified in this way for each classification group, it is possible to estimate the moving speed and moving direction of each part with respect to each classification group. Further, the position of each part can be estimated as the center of the polygon at all the feature points for each classification group.

  Next, processing in the moving object ID and information management unit 13 will be described. FIG. 5 is a flowchart showing the processing here.

  The moving object ID and information management unit 13 compares the information on the moving object area calculated by the moving object information calculation unit 12 with the registered information to determine whether the detected moving object area is for a new moving object. It is judged whether it is for a registered moving object or a moving object that is detected due to shielding between a plurality of moving objects, and if it is judged for a new moving object, the moving object region Information is newly registered with the new ID as information on the moving object, and if it is determined that it is for a registered moving object, the previous ID is taken over and the previously registered information is changed to newly calculated information. If it is determined that it is for a detected moving object due to shielding between multiple moving objects, it is separated into individual moving object information, and then the previous ID is taken over. Registration Information is updated to the new information. Note that, as shown in FIG. 3, the shielding occurs between a plurality of moving objects after an individual moving object is detected before that, so that information is updated here. I have to.

  For example, if the information on the moving objects a and b is already registered, and the moving object A is detected by the moving object detection unit 11, in S51 of FIG. Compared with the information of the objects a and b), the area coverage is calculated by the following equation (4), and it is determined whether or not it is equal to or less than the threshold value T1. Here, the comparison is performed after alignment according to the position of each moving object region.

  In S51, when it is determined that the coverage of the area of the moving object A calculated by the above equation (4) for all the areas of the already registered moving objects a and b is equal to or less than a preset threshold value T1, In S52, a new ID is assigned to the moving object A, and in S53, the moving object A and information on each part thereof are newly registered by the moving object ID and the moving object registration unit 16 of the information management unit 13. FIG. 6A shows the relationship between the moving object A and the registered moving object a in this case. As described above, when the area of the registered moving object a covered by the area of the moving object A is equal to or less than the threshold T1, the moving object A is given a new ID as a new moving object, and the moving object A and each part thereof Register new information.

  If it is determined in S51 that the area coverage ratio exceeds the threshold TI, the registered moving object covered by the area of the moving object A is selected as one reference moving object. In S54, it is determined whether the selected reference moving object includes only one registered moving object or a plurality of registered moving objects. If it is determined that the reference moving object includes only one registered moving object, The information on the selected registered moving object is updated with the information on the moving object A. Note that the ID takes over the ID of the registered moving object. FIG. 6B shows the relationship between the moving object A and the registered moving object a when the area of the moving object A covers only one registered moving object.

  If it is determined in S54 that the selected reference moving object includes a plurality of registered moving objects, the moving object A is a moving object that is detected by shielding between the plurality of moving objects. Proceeding to S56, the moving object separation unit 14 performs the shield moving object separation. Whether or not the reference moving object includes a plurality of registered moving objects can be determined from the moving object ID included in the reference moving object. FIG. 6C shows the relationship between the moving object A and the registered moving objects a and b when there are two registered moving objects covered by the area of the moving object A.

  In S56, information on each part of the moving object A calculated by the moving object information calculation unit 12, that is, the position of each part, its moving speed and moving direction, and a plurality of registered movements of the selected reference moving object The information of each part of the object is compared and matching processing is performed. In this matching process, one part of the moving object A is selected and compared with each part of all registered moving objects included in the reference moving object, and Diff = (position difference + movement speed difference + movement direction) The registered moving object part where the calculated Diff is the smallest value is set as a part corresponding to one part selected in the moving object A. Similarly, the other parts of the moving object A are selected one by one, and the part of the registered moving object corresponding to each part is searched.

  As described above, each part of the moving object A can correspond to a plurality of registered moving objects, so that the moving object A can be separated into each moving object according to the corresponding registered moving objects.

  FIG. 7 is an explanatory diagram of the separation of the occluding moving object.

  Here, as an example, the moving object A covers two registered moving objects a and b, and the area coverage is equal to or greater than the threshold T1, and the positions of the 18 parts A1-A18 of the moving object A and their movements It is assumed that the speed and direction of movement are calculated. Further, the positions of the nine parts a1-a9 of the registered moving object a and the moving speed and moving direction thereof are calculated, registered together with the moving object ID, and nine parts b1- of the registered moving object b. It is assumed that the position of b9 and its moving speed and moving direction are calculated and already registered together with the moving object ID.

  The order is arbitrary, but first, the above Diff is calculated between the part A1 of the moving object A and all the parts a1-a9, b1-b9 of the registered moving objects a, b, and the part corresponding to the part A1 Find a1. Similarly, the parts a2-a9, b1-b9 corresponding to the parts A2-A18 of the moving object A are searched for. Then, each part A1-A9 of the moving object A corresponding to each part a1-a9 of the registered moving object a is separated as one moving object, and the same moving object ID as that of the registered moving object a is assigned to the moving object. And each part A10-A18 of the moving object A corresponding to each part b1-b9 of the registered moving object b is separated as one moving object, and the same movement as the registered moving object b is performed on the moving object Give the object ID. Note that the position and size of the separated moving object may be calculated after the separation.

  After that, the process proceeds to S55, and the previous ID is taken over and the information on the registered moving objects a and b is separated from the moving object A as in the case where it is determined as one registered moving object in S54. Update with moving object information.

  Although the embodiment has been described above, the present invention is not limited to the above embodiment. For example, in the above description, the moving object is described as a person. However, the moving object may be an animal other than a person, a car, or the like, or a mixture of them.

  Further, the present invention can be realized not only as a moving object extraction apparatus, but also as a moving object extraction method that sequentially executes processing in each unit, or as a program that causes a computer to function as each unit of the moving object extraction apparatus.

10 ... moving object extraction device, 11 ... moving object detection unit, 12 ... moving object information calculation unit,
13 ... Moving object ID and information management unit, 14 ... Shield moving object separation unit, 15 ... Moving object update unit, 16 ... Moving object registration unit

Claims (7)

A moving object extraction device for extracting a moving object from a moving image,
A moving object detection unit for detecting a moving object region in a moving image;
A moving object information calculating unit that calculates information about the moving object region and each part that moves differently in the moving object region;
The information on the moving object area calculated by the moving object information calculation unit is individually compared with the information on all registered moving objects, and the moving object area corresponds to only one moving object or a plurality of movements. If it is determined whether or not the object corresponds to only one moving object, the information calculated by the moving object information calculation unit is newly registered as information on the moving object, or When the registered information is updated for the moving object and it is determined that the moving object corresponds to a plurality of moving objects, the information about each part calculated by the moving object information calculation unit is obtained for the plurality of moving objects. update registered to the mobile object from the separated information calculated by the moving object information calculation section compared for each site and the registered information in the information of each moving object Te Moving object extraction apparatus comprising the moving body information management unit. The moving object detector is
Detecting a motion vector and a feature point group in the moving vector from at least two frame images;
Classifying the feature point group according to the detected motion vector;
Extract the non-convex shape from the classified feature point group to generate the contour of the moving object region,
Estimate the approximate area of the moving object area from the generated contour,
The moving object extraction apparatus according to claim 1, wherein an approximate area of the estimated moving object area is used as a foreground, and the entire image is divided into a foreground and a background to extract the moving object area. The moving object information calculation unit
Calculate the geometric center of the moving object area as the position of the moving object area,
Calculate the number of pixels in the moving object area as the size of the moving object area,
Classify each part according to the speed and direction of the motion vector of each part of the moving object region, calculate the geometric center of each classified part as the position of each part,
The moving object extraction device according to claim 1 or 2, wherein a moving speed and a moving direction of each part are calculated from the motion vectors of the classified parts. The mobile information management unit
4. The moving object extraction apparatus according to claim 1, wherein an ID is assigned to each moving object and information thereof is managed. The mobile information management unit
The determination as to whether or not the moving object region corresponds to only one moving object or a plurality of moving objects from the overlap of the moving object region and the registered moving object region. 5. The moving object extraction device according to any one of 1 to 4. A moving object extraction method for extracting a moving object from a moving image,
Detecting a moving object region in the moving image;
Calculating information about the moving object region and each part that moves differently in the moving object region;
The information about the moving object area calculated in the step of calculating the information is individually compared with the information of all registered moving objects, so that the moving object area corresponds to only one moving object or a plurality of movements. If it is determined whether or not it corresponds to only one moving object, the information calculated in the step of calculating the information is newly registered as information on the moving object, or When it is determined that the registered information is updated with respect to the moving object and it corresponds to a plurality of moving objects, the information about each part calculated in the step of calculating the information is obtained for the plurality of moving objects. update registered to the mobile object from the separated information calculated in the step of calculating the information compared for each site and the registered information in the information of each moving object Te Moving object extraction method characterized by having a step. A program for extracting a moving object from a moving image, comprising:
Moving object detection means for detecting a moving object region in a moving image;
Moving object information calculating means for calculating information about the moving object region and each part that moves differently in the moving object region;
The information on the moving object area calculated by the moving object information calculation unit is individually compared with the information on all registered moving objects, and the moving object area corresponds to only one moving object or a plurality of movements. If it is determined whether or not the object corresponds to only one moving object, the information calculated by the moving object information calculation unit is newly registered as information on the moving object, or If the registered information is updated for the moving object and it is determined that it corresponds to a plurality of moving objects, the information about each part calculated by the moving object information calculating unit is given to the plurality of moving objects. update registered to the mobile object from the separated information calculated by the moving object information calculation section compared for each site and the registered information in the information of each moving object Te Program to function as the mobile object information management means that.