JP2009266155A - Apparatus and method for mobile object tracking - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a mobile object tracking apparatus which can track a mobile object with high accuracy without carrying out a complicated process. <P>SOLUTION: The mobile object tracking apparatus is provided with: at least two pieces of photographing means 1a and 1b which are in a known positional relationship; an image storing section 4 with a signal processing unit 3 for fetching a photographed image through a fetching means 2 at constant intervals and where the fetched image is stored; a corresponding point measuring means 5 for obtaining a corresponding point; a corresponding point detection determining means 6 for determining whether or not the corresponding point is correctly obtained; a three-dimensional position measuring means 7 for measuring a three-dimensional position of the mobile object in the image when the corresponding point is detected; a three-dimensional position estimating means 8 for waiting for a next image input when the corresponding point cannot be detected, checking whether the mobile object exists by using the images which are of before and after that time when the corresponding point could not be detected, and then estimating the three-dimensional position from the images before and after the image when existent; and a mobile object tracking means 9 for obtaining a track of the mobile object from the three-dimensional position obtained by the three-dimensional position measuring means or the three-dimensional position estimating means. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a moving body tracking apparatus and method for tracking a moving body even when information on the moving body of a captured image is lost.

  In general, as a method of tracking a moving object of this type, a wireless transmitter is provided for each moving object, and the moving object is tracked from wireless information, or the moving object is extracted from an image captured by an imaging device and tracked. Various methods have been devised, such as a method to do this.

  Among these, the tracking method using an image does not require a special device for the moving body, so it becomes possible to track a large number of unspecified moving bodies, and from images obtained simultaneously from a plurality of imaging devices. A method of tracking the moving body by measuring the three-dimensional position of the moving body has been devised.

  As such a mobile body tracking device, for example, as shown in Patent Document 1, a mobile body detection system, a mobile body detection device, a mobile body detection method, and a method for tracking a mobile body including a mobile body detection program are known. ing. In this Patent Document 1, the proportion of a moving object is measured from a stereo image, the template is changed according to the congestion level to facilitate the identification of the moving object, and the moving object is tracked by a method such as an optical flow. Is. Also, a technique is known in which when a moving body overlaps, a labeling process is performed with a plurality of moving bodies as one region, and the moving body is separated using a concept of a lap rate.

Moreover, as shown in Patent Document 2, there is known an apparatus for tracking a moving body called a pedestrian detection apparatus. In this patent document 2, the detection accuracy of a moving object is improved by separating a moving object (pedestrian) using height information among three-dimensional position information using a stereo image.
JP-A-2005-235104 JP 2002-24986 A

  In the conventional mobile tracking device described above, it is considered effective when three-dimensional information can be measured.

  However, there is a problem that the moving bodies cannot be separated because the moving bodies may overlap on the image and three-dimensional information may not be obtained.

  In addition, as shown in Patent Document 1, a method of processing a plurality of moving objects as one label and separating the moving objects has been proposed.

  However, depending on the movement of the moving body, there is a possibility of tracking a different moving body. In such a case, there are problems such as preparing a plurality of templates and requiring complicated processing.

  The present invention has been made to solve the above problems, and provides a moving body tracking apparatus and method for tracking a moving body with high accuracy without performing complicated processing by using time-series images. With the goal.

  In order to achieve the above object, the present invention provides a movement having at least two photographing means having a known positional relationship and a signal processing device in which an image photographed by the photographing means is captured via the image capturing means. In the body tracking device, the signal processing device and a corresponding point measuring unit that detects a corresponding point from the image captured via the image capturing unit, and a correspondence that determines whether or not the corresponding point is detected A point detection determination unit; a three-dimensional position measurement unit that measures a three-dimensional position of a moving object in the image when the corresponding point is detected; and a corresponding point that is not detected when the corresponding point cannot be detected. A three-dimensional position estimation means for estimating a three-dimensional position from the images before and after the detection failure, and the three-dimensional position obtained by the three-dimensional position measurement means or the three-dimensional position estimation means. And moving object tracking device that enables tracking of the moving object and is characterized in that it comprises.

  In order to achieve the above object, according to the present invention, in a moving body tracking method for tracking a moving body using images picked up by at least two image pickup means, information on the moving body is lost in the image. In this case, the moving object can be tracked using the images obtained before and after the time when this information is lost.

  According to the moving object tracking device and the method of the present invention, even when the moving object is hidden behind another moving object in one of the images captured by at least two imaging means, the time-series image is displayed. Can be used to estimate the three-dimensional position of the moving body and track the moving body with high accuracy.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of a mobile tracking device and method according to the present invention will be described below with reference to the drawings. Here, the same or similar parts are denoted by common reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram showing the configuration of the mobile tracking device according to the embodiment of the present invention.

  First, the configuration of the mobile tracking device will be described with reference to FIG.

  As shown in this figure, the moving body tracking device has two cameras 1a and 1b, which are imaging means 1 having a known positional relationship, and images taken by the cameras 1a and 1b at regular time intervals. A signal processing device 3 to be captured via the capturing means 2 is provided.

  The signal processing device 3 includes an image storage unit 4 that stores the captured image, a corresponding point measuring unit 5 that determines a corresponding point from the captured image, and a correspondence check that determines whether the corresponding point is accurately determined. An output determining means 6 and a three-dimensional position measuring means 7 for measuring the three-dimensional position of the moving body in the image when this corresponding point is detected are provided.

  In addition, when the corresponding point cannot be detected, the signal processing device 3 waits for the next image input, confirms whether a moving body exists using images before and after the corresponding point cannot be detected, If there is a three-dimensional position estimation means 8 for estimating the three-dimensional position from the preceding and following images, and the trajectory of the moving object from the three-dimensional position obtained by the above-described three-dimensional position measurement means 7 or the three-dimensional position estimation means 8. Mobile body tracking means 9 is required.

  In the present embodiment configured as described above, a method for obtaining position information using two cameras 1a and 1b will be described with reference to FIGS.

  FIG. 2 is a flowchart showing processing of the signal processing unit in FIG. 1, and FIG. 3 is an explanatory diagram showing an image of the captured image in FIG.

  First, the two cameras 1a and 1b which are the photographing means 1 are fixed and have a known positional relationship.

  Before starting the moving body tracking, camera parameters that associate the three-dimensional position and the position on the image are obtained for each of the cameras 1a and 1b. This camera parameter includes a test cube or the like having a plurality of known three-dimensional coordinates (xi, yi, zi) (i is a reference point number 1 to n, and at least 6 points are required). Shoot simultaneously using 1b. A test cube has a cube-shaped surface that has three corners orthogonal to each other and is polished with high precision. The prism and other components are brought into contact with this surface to increase the height of these components. It is a device used to detect pyramids, resolution, etc.

  Images simultaneously captured using the cameras 1a and 1b are set as a right image ImR and a left image ImL, respectively (S1).

  Pattern matching is performed on the images in the right image ImR and the left image ImL (S2), and corresponding points are obtained (S3).

  As an example, the points (u1i, v1i) and (u2i, v2i) on the image corresponding to the reference point i on the right image ImR and the left image ImL photographed simultaneously using the cameras 1a and 1b are obtained, and the following ( In the equations 1) and (2), camera parameters P1 and P2 are obtained by calculating camera parameters P that simultaneously satisfy i = 1 to n.

λ1 (u1i, v1i, 1) = P1 (xi, yi, zi, 1) (1)
λ2 (u2i, v2i, 1) = P2 (xi, yi, zi, 1) (2)
After obtaining the camera parameters P1 and P2 for the two cameras 1a and 1b, respectively, the cameras 1a and 1b are synchronized, and the right image ImR (t) and the left image ImL (t ) Is obtained (S3).

  Further, after the corresponding points of the right image ImR (t) and the left image ImL (t) are obtained by the signal processing device 3, the three-dimensional coordinates of the corresponding points are obtained by calculation (S4).

  The search for the corresponding point is performed by, for example, performing a difference process with an arbitrary 3 × 3 pixel in the whole of the other image on the basis of 3 × 3 pixels of one of the input images, and making the most difference. Corresponding points can be detected by obtaining a small number of 3 × 3 pixels. If this corresponding point can be detected, the three-dimensional position coordinates can be obtained by calculation using the camera parameter P obtained in advance (S4).

  As described above, if the camera parameters P1 and P2 are accurately obtained and the corresponding points between the images can be accurately detected, the three-dimensional position of the object on the image can be relatively easily determined using the two cameras 1a and 1b. Can be sought. Note that the above-described method of obtaining position information and the corresponding point search method are examples of a number of methods, and are not limited to this method.

  As a method for recognizing a moving object from a time-series image, there is a method for estimating a movement of a moving object by obtaining a movement vector using pattern matching called an optical flow. The optical flow is a method for detecting a moving object in a two-dimensional plane.

  Here, using the right image ImR (t−Δt) and the right image ImR (t), or the left image ImL (t−Δt) and the left image ImL (t), respectively, using the optical flow, the right image ImR or the left image A moving body of a time-series image in the image ImL is obtained. If it is determined that the point a in the right image ImR (t−Δt) has moved to the point b in the left image ImL (t−Δt), the points a and b are obtained using the above-described three-dimensional position coordinate method. If the three-dimensional position coordinate for is obtained, the movement locus for the point a can be obtained (S9).

  However, when obtaining a three-dimensional position or obtaining a movement destination in a time-series image, it may not be possible to accurately obtain a three-dimensional position or a movement destination depending on a captured image. For example, consider the case of tracking a person with reference to FIG. FIG. 3 is an explanatory diagram showing an image of the photographed image of FIG. 1, (a) is an image diagram of the right image and the left image, and (b) is an image diagram when it is determined that the corresponding point cannot be detected. .

  Consider the case where a person is tracked from an image showing a plurality of persons as shown in FIG.

  Until the time (t−Δt), the persons A and B appear in the right image ImR and the left image ImL, respectively, and corresponding points can be detected (S3). Accordingly, the three-dimensional positions of the persons A and B can be obtained, and the movement destination after the elapse of time in the right image ImR and the left image ImL can be measured. Can be tracked.

  At time t, in the right image ImR (t), both the persons A and B appear in the image, but in the left image ImL (t), the person A overlaps with the person B, and the person A is detected in the image. I can't. When obtaining the three-dimensional position, the corresponding point is automatically detected by performing the difference processing of 3 × 3 pixels as described above in the region where the person A is captured with the right image ImR as the reference image. Even if is large, the point with the smallest difference in the searched range is detected as the corresponding point, so that the corresponding point is detected at an inappropriate position. As a result, the three-dimensional position of the person A is obtained as an inappropriate position.

  Further, in the left image ImL (t−Δt) and the left image ImL (t), the movement destination is obtained as an inappropriate position. As a result, accurate tracking of the moving object may not be possible.

  Here, it is determined by the corresponding point detection determination unit which is the corresponding point detection determination unit 6 whether the corresponding point detected by the corresponding point detection unit which is the corresponding point measurement unit 5 of FIG. An example of a determination method for preventing inappropriate tracking of a moving object will be described below.

  Regarding this determination, for example, in the case of detecting a three-dimensional position using the right image ImR (t) and the left image ImL (t), as described above, in the case of performing the difference processing of 3 × 3 pixels, the corresponding points A difference occurs in the magnitude of the difference value depending on the presence or absence of.

  Here, a threshold value is set for the magnitude of the difference value, and if it is less than or equal to the threshold value, it is determined that the corresponding point exists normally. I do.

  If it is determined that the corresponding point is correctly detected, the three-dimensional position of the corresponding point is obtained using the coordinates of the corresponding point between the camera parameters P1 and P2 obtained in advance and the right image ImR and the left image ImL. The movement trajectory is obtained from the corresponding point with the time (t−Δt) in the right image ImR (or the right image ImL).

  If it is determined that the corresponding point has not been correctly detected, the process for obtaining the three-dimensional position at the time t is not performed, the image input at the next time (t + Δt) is waited, and the time (t−Δt) and time t The presence / absence of a moving object and the three-dimensional position at time t are estimated using the three-dimensional position estimation means 8 in FIG. 1 using time (t + Δt).

  Next, a process when it is determined that the corresponding point has not been detected (S5) will be described below.

1. Check for moving objects:
Corresponding point search is performed for each of the right image ImR and the right image ImL at time (t−Δt) and time t. Here, only the presence or absence of corresponding points is detected. If there is a corresponding point in either the right image ImR or the left image ImL, it is determined that there is a moving object (S6). If there is no corresponding point in both images, it is determined that there is no moving object (S7).

2. Position estimation of moving object:
If it is determined that a moving object is present, the time of time (t−Δt) and the time (t + Δt) using the right image ImR and the left image ImL, respectively, as in the method for obtaining the movement locus at the time t, A movement trajectory at time (t + Δt) is obtained from (t−Δt), and the three-dimensional position at time t is estimated by, for example, linear interpolation (S8).

  The above 1. In “Confirmation of presence / absence of moving object”, if there is no corresponding point in both images, it is determined that there is no moving object immediately, and further, moving using time (t + 2Δt), time (t + 3Δt), and the image after the elapse of time. The presence or absence of a body can be determined, and the position of the moving body at time (t + Δt) and time (t + 2Δt) can also be estimated. As described above, the moving object tracking apparatus can track the moving object using the time series data before and after the moving object overlaps on the image.

  According to the present embodiment, even when one of the images taken by the two camera cameras 1a and 1b is hidden behind another moving body and cannot be seen, it is moved using a time-series image. It is possible to estimate the position of the body. Thus, even when no corresponding point is obtained on the image, the three-dimensional position can be estimated using the time-series image, so that the moving object can be tracked with high accuracy.

  Next, a mobile tracking device according to a second embodiment of the present invention will be described. The configuration of the present embodiment is the same as that of FIG. 1, and a part of the function of the signal processing device 3 is different.

  Similar to the moving body tracking apparatus of the first embodiment, the moving body up to time t is tracked, and the moving area at time (t + Δt) is estimated. Here, a moving body when a pedestrian or the like moves at a constant speed is assumed.

  For example, it is assumed that at time t = (t−Δt) and time t, it is measured that the moving body exists at (x, y) and (x + Δx, y + Δy), respectively. In this case, assuming that the position of the moving body at t = (t + Δt) has moved linearly, it can be estimated that the position is (x + 2Δx, y + 2Δy). Limited to.

  According to the present embodiment, the processing time can be increased by limiting the region where pattern matching is performed.

  In this embodiment, it is assumed that the moving body moves linearly, but it is also possible to estimate the position of the moving body more accurately by secondary fitting or the like using the past three or more points. Is possible.

  The embodiments of the present invention have been described above. However, the present invention is not limited to the embodiments described above, and departs from the gist of the present invention by combining the configurations of the embodiments. Various modifications can be made without departing from the scope.

The block diagram which shows the structure of the mobile tracking device of embodiment of this invention. The flowchart which shows the process of the signal processing apparatus of FIG. 2A and 2B are explanatory diagrams showing images of the photographed image of FIG. 1, in which FIG. 1A is an image diagram of the right image and the left image, and FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Imaging | photography means, 1a, 1b ... Camera, 2 ... Image capture means, 3 ... Signal processing apparatus, 4 ... Image storage part, 5 ... Corresponding point measurement means, 6 ... Corresponding point detection determination means, 7 ... Three-dimensional position measurement Means 8 ... Three-dimensional position estimation means 9 ... Moving body tracking means 10 ... Moving body, A, B ... Person, ImR ... Right image, ImL ... Left image.

Claims (3)

In a mobile tracking device having at least two imaging means having a known positional relationship, and a signal processing device in which an image captured by the imaging means is captured via the image capturing means,
The signal processing device is
Corresponding point measuring means for detecting corresponding points from the image captured via the image capturing means;
Corresponding point detection determining means for determining whether or not this corresponding point is detected;
3D position measuring means for measuring the 3D position of the moving body in the image when the corresponding point is detected;
If the corresponding point cannot be detected, a three-dimensional position estimating means for estimating a three-dimensional position from the images before and after the corresponding point could not be detected;
Moving body tracking means capable of tracking the moving body from the three-dimensional position obtained by the three-dimensional position measuring means or the three-dimensional position estimating means;
A moving body tracking device comprising: In a moving body tracking method for tracking a moving body using images picked up by at least two image pickup means,
A method of tracking a moving object, wherein when the information on the moving object is lost in the image, the moving object can be tracked using the images obtained before and after the time when the information is lost.   The position of the said mobile body in the next time is estimated from the locus | trajectory of the said mobile body to a certain time, and the area | region which performs pattern matching is limited to the surrounding area | region of the said mobile body expected. Moving body tracking method.

Images