JP2008033818A - Object tracking device and its control method, object tracking system, object tracking program, and recording medium recording the program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object tracking device or the like capable of realizing speed improvement of object tracking processing. <P>SOLUTION: The object tracking device 3 is provided with a storing part 9 storing stereoscopic-photographed image from photographing equipment 2 via an image input part 7, and an object tracking part 8 tracking an object in stereoscopic-photographed images on the basis of a plurality of stereoscopic-photographed images at different points of time. The object tracking part 8 is provided with a characteristic point extracting part 11 using the stereoscopic-photographed images to extract a characteristic point and calculate an actual position of the extracted characteristic point, a correlation window setting part 13 setting an area on the photographed image being an area on the photographed image including the extracted characteristic point and having a size calculated on the basis of the calculated actual position of the characteristic point as an image of an object correlation window, and a collating part 14 uniforming sizes of the set image of the object correlation window and a stored image of a reference correlation window and carrying out collation. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an object tracking device that tracks an object in the captured image based on a plurality of captured images captured at different times, a control method thereof, an object tracking system, an object tracking program, and a record in which the program is recorded It relates to the medium.

  2. Description of the Related Art Conventionally, an object tracking system for tracking a captured object by capturing the motion of an object such as a vehicle, a person, or an object as a moving image by a camera and analyzing the moving image has been provided. 1-3. By tracking an object, the number of objects passing through a predetermined area can be measured, and the speed and flow line of the object can be grasped.

  Furthermore, the object tracking system can be used for applications such as traffic control and automobile driving assistance by recognizing movement of an automobile, for example. In addition, for example, by recognizing movements of passers-by in public places and customers in stores, it can be used for purposes such as traffic situation surveys and customer behavior surveys. Further, for example, by recognizing the movement of the production object in the production line of the factory and the movement of the worker, it can be used for applications such as production management and production planning.

  In such an object tracking system, an object is tracked by associating the same object in a captured image between a plurality of captured images captured at different times. For example, Patent Document 1 discloses that an object can be tracked by measuring a movement of a target object in a time-series image in accordance with a density change of the time-series image obtained by a video camera, a weather radar device, or the like. It is disclosed.

  In the object tracking device described in Patent Document 2, an object is imaged by a plurality of cameras, the distance to the object is measured by stereo image processing from the captured images, and the object is detected using the measured distance. Detected. Then, feature quantities such as luminance information of the image are extracted as templates from the detected object region, and matching is performed to track the object.

Further, in the forward vehicle tracking system described in Patent Document 3, the front of the host vehicle is scanned using a laser radar to detect the forward vehicle and its position and measure the distance to the forward vehicle. . When a preceding vehicle is detected, a tracking template for the preceding vehicle is created at a position on the captured image corresponding to the detected position of the preceding vehicle, and the size change due to the distance change is taken into account from the created tracking template. The vehicle is tracked by performing template matching.
JP 10-197543 A (published July 31, 1998) JP 11-252587 A (published September 17, 1999) JP 2004-53278 A (published February 19, 2004)

  When the object tracking process as described above is processed in real time, it is necessary to shorten the processing time. For example, when the movement of an object is fast, such as a vehicle, it is necessary to increase the number of frames per unit time in a moving image. In this case, the time between frames is shortened, and this short frame It is necessary to complete the association within the time period.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an object tracking device and the like that can realize high-speed object tracking processing.

  In order to solve the above-described problem, the object tracking device according to the present invention includes an image acquisition unit that acquires a plurality of captured images created by capturing at different points in time, and the imaging based on the acquired plurality of captured images. An object tracking device comprising an object tracking means for tracking an object in an image, wherein a reference correlation window is used to refer to a partial region of the captured image including feature points that characterize a partial shape of the object in a captured image obtained by past imaging The image acquisition unit further includes a storage unit that stores image information of the reference correlation window, and the image acquisition unit captures a stereo captured image, which is a plurality of captured images simultaneously captured from different positions, at a certain point in time. And the object tracking means uses at least two captured images among the stereo captured images acquired by the image acquiring means, A feature point extracting means for extracting the feature point and calculating actual position information, which is information relating to the actual position of the extracted feature point, and a part of the photographed image including the feature point extracted by the feature point extracting means A correlation window setting unit that sets a partial region of the captured image having a size calculated based on the actual position information of the feature point calculated by the feature point extraction unit as a target correlation window; The image processing apparatus is characterized by further comprising collating means for collating the image of the target correlation window set by the correlation window setting means and the image of the reference correlation window stored by the storage means with the same size.

  The object tracking device control method according to the present invention acquires a plurality of captured images created by capturing images at different times, and tracks an object in the captured image based on the acquired captured images. An object tracking device that stores image information of a reference correlation window using a partial area of the captured image including a feature point that characterizes a partial shape of an object in a captured image obtained by past imaging as a reference correlation window. An object tracking device control method comprising a storage means, in order to solve the above problems, an image acquisition step for acquiring a plurality of captured images created by shooting at different points in time, and a plurality of acquired captured images An object tracking step of tracking an object in the captured image based on the image acquisition step, wherein the image acquisition step includes a plurality of captured images captured simultaneously from different positions. And the object tracking step uses at least two of the stereo captured images acquired in the image acquiring step. A feature point extracting step of extracting the feature point and calculating actual position information that is information on an actual position of the extracted feature point, and the feature point extracted in the feature point extracting step. A partial area of the photographed image having a size calculated based on the actual position information of the feature point calculated in the feature point extraction step is set as a target correlation window. The correlation window setting step, the image of the target correlation window set in the correlation window setting step, and the reference correlation window image stored in the storage means are collated with the same size. It is characterized in that it comprises a verification step that.

  Here, the captured image may be a color image or a black and white image including only luminance information as long as the image can be traced. The captured image may be an image that has been subjected to image processing for facilitating tracking of an object, such as sharpening processing.

  Moreover, as an example of the feature point, there is a portion where the luminance changes rapidly in the captured image or the processed image. Examples of the actual position information include information on the distance from the photographing device that performs photographing to the feature point, position information on coordinates set in advance in the photographing space, and the like.

  According to the above configuration or method, feature points are extracted and actual position information of the extracted feature points is calculated using at least two captured images among stereo captured images. As a result, the object specifying process can be omitted as compared with the conventional case in which the object on the image is specified from the feature points and the actual position information of the specified object is calculated. Further, the number of times of calculating the actual position information can be reduced as compared with the conventional case where the captured image or the processed image is divided into a large number of small areas and the actual position information is calculated for each divided small area. Therefore, compared with the conventional case, it is possible to speed up the processing until the actual position information of the feature point is calculated from the captured image.

  Further, a target correlation window including the extracted feature points is set, and the set target correlation window image and the reference correlation window image are collated. Thereby, compared with the conventional case where template matching is performed on an image to be searched using a template including an object, the size to be compared can be reduced, and the number of comparisons can be significantly reduced. As a result, the collation process can be speeded up.

  Therefore, the object tracking device and the control method for the device according to the present invention can realize speeding up of the object tracking processing.

  Further, the size of the target correlation window including the extracted feature point is determined based on the actual position information of the feature point. Thereby, even if the size of the object on the captured image changes as the object approaches or moves away from the imaging device, the size of the target correlation window can be changed in the same manner. Therefore, the size of the reference correlation window and the target correlation window can be set to a size suitable for the partial shape of the object corresponding to the feature point, so that the accuracy of matching between the reference correlation window image and the target correlation window image is increased. Can be further improved.

  Note that the image acquisition unit or the image acquisition process may acquire a plurality of captured images captured at different times as a moving image or intermittently as a still image.

  In the object tracking device according to the present invention, it is preferable that the collating unit collates the image of the target correlation window and the image of the reference correlation window by reducing the larger size to the smaller size. In this case, since the images are collated with the smaller size, the amount of calculation required for the collation processing can be suppressed. Therefore, it is possible to further speed up the object tracking process.

  Incidentally, interpolation processing is usually performed when an image is enlarged or reduced. At this time, when the image is reduced, it is less likely that erroneous information of the image is mixed by the interpolation process than when the image is enlarged. Therefore, according to the above configuration, the accuracy of matching can be further improved, and as a result, the object can be tracked with high accuracy.

  In the object tracking device according to the present invention, the object tracking means further includes search range setting means for setting, as a search range, a range in which the feature point can move based on the position of the feature point included in the reference correlation window. The collating unit preferably collates the image of the target correlation window including a feature point within the search range set by the search range setting unit and the image of the reference correlation window.

  Here, the range in which the feature point can move is set using, for example, the traveling direction and speed of the feature point, the maximum speed at which the object corresponding to the feature point can reach, the restricted traveling direction with respect to the object, and the like. be able to. Further, in the case where the collation process is executed for each frame and the object does not move so much between the frames, a range in which the feature point can move within a predetermined range from the feature point may be set.

  According to the above configuration, the target correlation window to be collated is limited to those having feature points within the search range, so that the number of times of collating the image of the target correlation window with the image of the reference correlation window can be reduced. it can. Therefore, it is possible to further speed up the object tracking process.

  By the way, when a stereo photographed image is used, a conversion process between a position on the photographed image and an actual position can be easily executed. Therefore, the search range may be a range of positions on the image, or may be a range of actual positions.

  Further, the feature point extracting means may extract only the feature points within the search range, or the correlation window setting means may set the target correlation window only for the feature points within the search range. In these cases, the number of feature point extraction processes and target correlation window setting processes can be reduced, so that the object tracking process can be further accelerated.

  By the way, the size of the object in the captured image and the imaging distance from the imaging device that performs imaging to the object are in an inversely proportional relationship. That is, the size of the partial shape of the object in the photographed image and the photographing distance from the photographing device to the feature point characterizing the partial shape are in an inversely proportional relationship.

  Therefore, the actual position information is information on the shooting distance from the shooting device that performs the shooting to the feature point, and the storage unit applies the shooting distance information on the feature point included in the reference correlation window. The correlation window setting means relates to a feature point included in the target correlation window: Equation (size of the reference correlation window) = (size of the target correlation window) × It is preferable to determine the size of the reference correlation window based on (the shooting distance) / (the shooting distance regarding the feature points included in the reference correlation window).

  By the way, the shape of the object on the captured image changes with time. For this reason, as the period between the shooting time of the shot image corresponding to the reference correlation window and the shooting time of the shot image corresponding to the target correlation window becomes longer, the accuracy of the collation decreases.

  In view of this, it is preferable that the collating unit stores information on the target correlation window whose image is most similar to the reference correlation window in the storage unit as information on a new reference correlation window. In this case, since collation can be performed using a reference correlation window related to a recent photographed image, it is possible to prevent degradation of collation accuracy. Note that the information on the new reference correlation window may be stored in the storage unit so as to be added to the information on the existing reference correlation window, or may be stored in the storage unit so as to be updated.

  In the object tracking device according to the present invention, it is preferable that the storage unit stores information related to the shooting time point when the shot image corresponding to the reference correlation window is shot in association with the reference correlation window. In this case, since the period can be calculated from the shooting time of the shot image corresponding to the reference correlation window to the shooting time of the shot image corresponding to the target correlation window, the speed of the feature point can be calculated. Note that the information related to the shooting time includes the serial number of the frame in addition to the shooting time information.

  By the way, the number of feature points included in the captured image may be one or plural. When a plurality of feature points are included in a captured image, it is desirable that individual feature points can be tracked. Therefore, it is preferable that the storage unit stores information regarding a plurality of reference correlation windows regarding different feature points. In this case, the matching process is performed for each reference correlation window.

  In addition, the imaging device provided with the 1st imaging | photography part and the 2nd imaging | photography part which image | photograph simultaneously from a mutually different position, and the some imaging | photography image | photographed from the imaging | photography apparatus at different time points are acquired, If the object tracking system includes the object tracking device configured as described above that tracks an object in the captured image based on the image, the same effects as described above can be obtained.

  The object tracking means in the object tracking device can be made to function on a computer by an object tracking program. Furthermore, the object tracking program can be executed on an arbitrary computer by storing the object tracking program in a computer-readable recording medium.

  As described above, the object tracking device according to the present invention extracts feature points, sets a target correlation window including the extracted feature points, and collates the set target correlation window image with the reference correlation window image. Therefore, the process of specifying the object can be omitted, and therefore the object tracking process can be speeded up. Further, by determining the size of the target correlation window including the extracted feature point based on the actual position information of the feature point, the size of the reference correlation window and the target correlation window is set to a part of the object corresponding to the feature point. Since the size can be adapted to the shape, it is possible to further improve the accuracy of matching between the image of the reference correlation window and the image of the target correlation window.

[Embodiment 1]
An embodiment of the present invention will be described with reference to FIGS.

(Configuration of object tracking system)
FIG. 1 shows a schematic configuration of an object tracking system 1 according to the present embodiment. Although the object tracking system 1 of this embodiment tracks a vehicle traveling on a road and measures the number and speed of the vehicles, the present invention is not limited to this. As illustrated, the object tracking system 1 is configured to include an imaging device 2, an object tracking device 3, and a display device 4.

  The photographing device 2 is installed above a road and photographs a predetermined area of the road from a bird's-eye view. The imaging device 2 includes a first imaging unit 5 and a second imaging unit 6 that simultaneously capture a predetermined area of a road from different positions. The first imaging unit 5 and the second imaging unit 6 are configured by an imaging element such as a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor) imaging element, for example. The first imaging unit 5 and the second imaging unit 6 transmit time-series captured image data that has been captured to the object tracking device 3.

  In the present embodiment, the imaging device 2 performs video shooting and transmits the captured moving image data to the object tracking device 3 as time-series captured image data. Hereinafter, two sets of moving image data captured by the first imaging unit 5 and the second imaging unit 6 are collectively referred to as “stereo moving image data”. Of the two sets of moving image data, two sets of still image data in a certain frame are collectively referred to as “stereo shot image data”.

  The imaging device 2 may be configured by a stereo camera integrally including the first imaging unit 5 and the second imaging unit 6, or may be configured by two cameras as the first imaging unit 5 and the second imaging unit 6. May be. When the imaging device 2 is configured by two cameras, the first imaging unit 5 and the second imaging unit 6 assume that various parameters necessary for the imaging are calibrated in advance. Further, it is assumed that the distance between the cameras is known and stored in the object tracking device 3.

  The imaging device 2 may be configured to include three or more imaging units. When the number of photographing units is increased, real position information that is information related to the actual position of the subject can be obtained with high accuracy. In addition, the imaging device 2 may intermittently transmit still images and intermittently transmit still image data to the object tracking device 3. In addition, when the interval at which the still photographing is performed changes, it is necessary to transmit photographing time data to the object tracking device 3.

  The object tracking device 3 tracks a vehicle included in the captured image based on a plurality of captured images created by the imaging device 2 capturing images at different times. The object tracking device 3 transmits the tracking result data to the display device 4.

  In the object tracking device 3 according to the present embodiment, the feature points characterizing the partial shape of the object in the photographed image are extracted using the stereo photographed image acquired from the photographing device 2, and the actual positions of the extracted feature points are (Actual position) is calculated. Thereby, compared with the conventional case where an object on an image is specified from feature points and the actual position of the specified object is calculated, the process of specifying the object can be omitted. Further, the number of times of calculating the actual position can be reduced compared to the case where the captured image is divided into a large number of small areas and the actual position is calculated for each of the divided small areas. Therefore, the object tracking device 3 of the present embodiment can speed up the processing until the actual position of the feature point is calculated from the captured image, as compared with the conventional case.

  In the object tracking device 3 of the present embodiment, a target correlation window including the extracted feature points is set, and the set target correlation window image and the reference correlation window image are collated. As a result, the size to be compared can be reduced and the number of comparisons can be significantly reduced as compared with the case where template matching is performed on an image to be searched using a template including an object. As a result, the collation process can be speeded up.

  Therefore, the object tracking device 3 of the present embodiment can realize speeding up of the object tracking process. The detailed configuration of the object tracking device 3 will be described later.

  The display device 4 displays the result of tracking the vehicle by the object tracking device 3, and is configured by various display devices capable of displaying an image, such as a CRT (Cathode Ray Tube) or a liquid crystal display element. .

  Next, details of the object tracking device 3 will be described. The object tracking device 3 includes an image input unit (image acquisition unit) 7, an object tracking unit (object tracking unit) 8, a storage unit (storage unit) 9, and a display control unit 10.

  The image input unit 7 receives stereo moving image data captured by the first imaging unit 5 and the second imaging unit 6 of the imaging device 2 from the imaging device 2. Stereo moving image data received by the image input unit 7 is stored in the storage unit 9.

  The storage unit 9 stores various types of information, and includes a storage device such as an HDD (Hard Disk Drive) or a flash memory. In the present embodiment, the storage unit 9 stores stereo moving image data from the imaging device 2. In addition, the storage unit 9 stores image data and size data of the reference correlation window and shooting distance data from the imaging device 2 to the feature point corresponding to the reference correlation window in association with each other.

  The object tracking unit 8 uses the stereo moving image data and the reference correlation window image data stored in the storage unit 9 to track the vehicle in the moving image. Specifically, the object tracking unit 8 tracks feature points that characterize the partial shape of the vehicle in the moving image. The object tracking unit 8 includes a feature point extraction unit (feature point extraction unit) 11, a correlation window setting unit (correlation window setting unit) 13, and a collation unit (collation unit) 14.

  The feature point extraction unit 11 reads out the stereo shot image data of the latest frame from the stereo moving image data stored in the storage unit 9 from the storage unit 9 and uses the read out stereo shot image data in the shot image. The feature points are extracted, and the actual position information of the extracted feature points is calculated. Here, the actual position information is information related to the actual position, and includes, for example, three-dimensional information at the actual position of the feature point, a shooting distance from the image capturing device 2 to the actual position of the feature point, and the like. The feature point extraction unit 11 transmits the coordinate data in the captured image and the data of the actual position information to the correlation window setting unit 13 for each extracted feature point. The details of the feature point extraction method and the actual position information calculation method will be described later.

  The correlation window setting unit 13 sets, as a target correlation window, a small area on a captured image including feature points or a small area on a processed image obtained by processing a captured image, such as a grayscale image. The correlation window setting unit 13 transmits the set image data of the target correlation window to the matching unit 14.

  In the present embodiment, the correlation window setting unit 13 determines the size of the target correlation window based on the actual position information data of each feature point from the feature point extraction unit 11, and sets the target correlation window. Thereby, even if the size of the object on the captured image changes as the object approaches or moves away from the imaging apparatus 2, the size of the target correlation window can be changed in the same manner. Therefore, the size of the reference correlation window and the target correlation window can be set to a size suitable for the partial shape of the object corresponding to the feature point, so that the accuracy of matching between the reference correlation window image and the target correlation window image is increased. Can be further improved.

  The collation unit 14 collates the image of the target correlation window acquired from the correlation window setting unit 13 and the image of the reference correlation window acquired from the storage unit 9. Thereby, the collation part 14 can match | combine a target correlation window with a reference correlation window. That is, the feature points in the captured image at different times can be associated with each other. Details of the verification will be described later.

  If the size of the target correlation window is different from the size of the reference correlation window, it is difficult to perform the above collation. Therefore, the collation unit 14 performs the collation after aligning the sizes of the reference correlation window and the target correlation window.

  When the collation unit 14 associates the target correlation window with the reference correlation window as a result of the collation, the collation unit 14 transmits the coordinate data of the feature points included in the associated target correlation window to the display control unit 10. Furthermore, the collation unit 14 stores the image data of the associated target correlation window as new reference correlation window image data in the storage unit 9 together with the time or frame number data related to the frame. The image data of the target correlation window may be stored in the storage unit 9 so as to update the image data of the previous reference correlation window.

  The display control unit 10 controls the display image displayed by the display device 4. Specifically, the display control unit 10 displays the latest frame captured image stored in the storage unit 9 and displays information indicating the movement of the feature point based on the coordinate data from the matching unit 14. Control the display device. Thereby, the user can grasp the amount of movement of the object.

  The display control unit 10 prints the information indicating the movement of the feature points on the printing medium via the printing device instead of outputting the information to the display device 4, records the information on the recording medium via the recording device, It may be transmitted to another device via a communication device and a communication medium.

(Flow of object tracking process)
FIG. 2 shows a flow of object tracking processing in the object tracking system 1 having the above-described configuration. As shown in the figure, first, in step 101 (hereinafter referred to as S101), the photographing apparatus 2 photographs a predetermined area of the road, so that one frame of the stereo photographed image data is obtained from the photographing apparatus 2. Acquired by the image input unit 7 of the tracking device 3. The acquired one-frame stereo image data is stored in the storage unit 9.

  Next, the feature point extraction unit 11 extracts the feature point in the stereo shot image using the data of the stereo shot image stored in the storage unit 9 and acquires the three-dimensional information of the extracted feature point. (S102). FIG. 3 shows a flow of processing in which the feature point extraction unit 11 extracts feature points, and FIGS. 4A to 4C show images acquired in the process of extracting feature points. Yes.

  First, the data of the stereo image of the latest frame is acquired from the storage unit 9 (S150). FIG. 4A shows any one captured image among stereo captured images composed of stereo captured image data acquired from the storage unit 9.

  Next, edge extraction processing is performed from the stereo image (S151). FIG. 4B shows an image obtained by performing edge extraction processing on the photographed image shown in FIG. As shown in the drawing, an edge whose luminance change is equal to or greater than a predetermined value is extracted by the edge extraction process.

  The edge extraction process preferably extracts a horizontal edge when the object is a car, and preferably extracts an edge of a corner portion when the object is a person. Moreover, as a filter which extracts an edge, well-known filters, such as a Sobel filter, a Harris filter, a Laplacian filter, can be used, for example.

  Next, the three-dimensional information of the edges extracted from each of the stereo photographed images is calculated using the stereo method (S152). The height of the edge from the road can be obtained from the calculated three-dimensional information of the edge. The stereo method is described in publicly known literature (for example, Xugang, Saburo Saburo, “3D Vision”, Kyoritsu Publishing Co., Ltd., April 20, 1998, first edition). Omitted.

  Next, an edge whose height from the road is within a predetermined range is extracted as a feature point (S154). Here, the predetermined range is a range in which the edge is determined to have a high possibility of being derived from an automobile, for example, a range that is higher than a road surface or a floor and lower than a signboard or a ceiling. It is. FIG. 4C shows a state in which only the edges that are highly likely to be automobile edges are left among the plurality of extracted edges. This remaining edge is extracted as a feature point. After step 154, the feature point extraction process ends, and the process returns to the original routine. As shown in FIG. 4C, when there are a plurality of feature points, the following processing is performed for each feature point.

  Returning to FIG. 2, the correlation window setting unit 13 sets a target correlation window related to the feature points extracted by the feature point extraction unit 11 (S103). At this time, the size of the target correlation window to be set is determined based on the photographing distance from the photographing device 2 to the actual position of the feature point. This shooting distance can be obtained when the three-dimensional information of the edge is calculated using the stereo method in step 152 shown in FIG.

  Details of the target correlation window setting process performed by the correlation window setting unit 13 will be described with reference to FIGS. FIG. 5 shows the relationship between the camera coordinates (X, Y, Z) indicating the three-dimensional position of the object with reference to the photographing apparatus 2 and the image coordinates (x, y) indicating the position of the object on the image. Show. With respect to an object, the correspondence relationship between the actual three-dimensional position (X, Y, Z) and the position (x, y) on the image can be expressed by the following equation. It is assumed that the focal length of the photographing apparatus 2 is f and the scalar coefficient is s.

When formula (1) is expanded, x = f × X / Z and y = f × Y / Z. That is, the size of the object in the image is inversely proportional to the shooting distance Z from the shooting device 2 to the object. Therefore, the correlation window setting unit 13 stores the size data and shooting distance data associated with the image data of the reference correlation window read from the storage unit 9 and the shooting distance data of the feature points extracted by the feature point extraction unit 11. And the size of the target correlation window is calculated by the following equation.
(Size of target correlation window) = (Size of reference correlation window) × (Shooting distance related to feature points included in reference correlation window) / (Shooting distance related to feature points extracted by feature point extraction unit) (2) .

  Then, the correlation window setting unit 13 sets the target correlation window including the feature points extracted by the feature point extraction unit 11 with the size calculated from Expression (2).

  FIG. 6 shows one of the stereo captured images in the previous frame, and the reference correlation window RW set for the captured image. FIGS. 7A and 7B show captured images in the latest frame as shown in FIG. Among these, FIG. 7A is an example of the present embodiment, and shows a case where the target correlation window TW is set with the size calculated from the equation (2). FIG. 7B is a comparative example of the present embodiment and shows a case where the target correlation window TW ′ is set with the same size as the reference correlation window RW. Note that only a part of the target correlation windows TW and TW ′ set by the correlation window setting unit 13 is illustrated.

  When the captured image of the previous frame shown in FIG. 6 and the latest captured image of the frame shown in FIGS. 7A and 7B are compared, the vehicle is close to the imaging device 2 side. It can be understood that the size of the vehicle image increases. For this reason, as shown in FIG. 7B, when the size of the reference correlation window RW and the size of the target correlation window TW ′ are the same, the partial shape of the vehicle included in the reference correlation window RW is the target. There is a risk of protruding from the correlation window TW ′. At this time, it is difficult to accurately check the reference correlation window RW and the target correlation window TW ′.

  In order to avoid this problem, it is conceivable to increase the sizes of the reference correlation window RW and the target correlation window TW ′. However, in this case, the number of pixels to be collated increases and the amount of calculation increases. As a result, the object tracking process is delayed.

  On the other hand, in this embodiment, since the target correlation window TW is set with the size calculated from the equation (2), as shown in FIG. 7A, the portion of the vehicle included in the reference correlation window RW. The target shape is included in the target correlation window TW without excess or deficiency. Therefore, it is possible to accurately check the reference correlation window RW and the target correlation window TW.

  By the way, in the conventional template matching, it is conceivable to enlarge or reduce the template according to the shooting distance. FIG. 8 shows a case where template matching is performed using the image of the reference correlation window RW shown in FIG. 6 as a template as a comparative example. In this case, as shown in the figure, the photographed image is scanned and compared with the template, and a part on the photographed image similar to the template is specified.

  However, in the above case, it is necessary to calculate shooting distances at all positions to be scanned in order to enlarge or reduce the template. For this reason, the amount of calculation increases remarkably, and as a result, the object tracking process is delayed.

  On the other hand, in the present embodiment, only the shooting distance at the position where the feature point exists is calculated, and the target correlation window TW corresponding to the shooting distance is set as shown in FIG. Compared with the template matching described above, the amount of calculation is small, and as a result, the object tracking process can be speeded up.

  Returning to FIG. 2, the collation unit 14 collates the image of the target correlation window set by the correlation window setting unit 13 with the image of the reference correlation window stored in the storage unit 9 (S104). Thereby, the collation part 14 can identify the image of the target correlation window most similar to the image of the reference correlation window, and can associate the identified target correlation window with the reference correlation window. That is, the feature points in the captured image at different times can be associated with each other.

  As an example of this collation, the size of the reference correlation window and the size of the target correlation window are matched, and the similarity is calculated using a normalized cross-correlation method such as the following equation. In this case, the target correlation window with the highest similarity is associated with the reference correlation window.

Here, the similarity is S, the luminance information of the reference correlation window having the same size is G (t), the luminance information of the target correlation window having the same size is G (t + Δt), and the sizes are the same. The size of the reference correlation window and the target correlation window is m × n. Note that time t indicates the time when the captured image related to the reference correlation window is captured, and time (t + Δt) indicates the time when the captured image regarding the target correlation window is captured.

  Further, as another example of the above collation, the size of the reference correlation window and the size of the target correlation window are aligned, and the sum of the absolute values of the corresponding pixel value differences (with respect to the target correlation window and the reference correlation window) ( (Absolute difference sum) is obtained. In this case, the target correlation window with the smallest absolute difference sum becomes the target correlation window with the highest degree of similarity, so that the corresponding target correlation window is associated with the reference correlation window.

  Next, the display control unit 10 displays the captured image of the latest frame stored in the storage unit 9 and displays the target correlation window associated with the reference correlation window as a result of the collation by the collation unit 14. The display device is controlled (S105). Thereby, the user can grasp the amount of movement of the object. Then, it returns to step 101 and repeats the said operation | movement.

[Embodiment 2]
Next, another embodiment of the present invention will be described with reference to FIGS. For convenience of explanation, the same reference numerals as those in the above embodiment are used for the same elements and processes as those in the above embodiment, and the description thereof is omitted. Further, the object tracking system of this embodiment is different from the object tracking system 1 shown in FIG. 1 only in the object tracking unit of the object tracking apparatus, and the other configurations are the same. Therefore, in the present embodiment, the configuration and processing of the object tracking unit will be mainly described.

  FIG. 9 shows a schematic configuration of the object tracking system 20 according to the present embodiment. As illustrated, the object tracking system 20 is configured to include an imaging device 2, an object tracking device 21, and a display device 4.

  The object tracking device 21 includes an image input unit 7, an object tracking unit 22, a storage unit 9, and a display control unit 10. The object tracking unit 22 tracks the feature points of the vehicle in the moving image by using the stereo moving image data and the reference correlation window image data stored in the storage unit 9. The object tracking unit 22 is different from the object tracking unit 8 shown in FIG. 1 in that an image reduction unit (collation unit) 23 is added between the correlation window setting unit 13 and the collation unit 14. The configuration is the same.

  The image reduction unit 23 reduces the larger one of the target correlation window image set by the correlation window setting unit 13 and the reference correlation window image stored in the storage unit 9 to the smaller one. is there. Therefore, the collation unit 14 collates the images of the target correlation window and the reference correlation window that are aligned in the smaller size.

  FIG. 10 shows the flow of an object tracking process in the object tracking system 20 having the above configuration. As shown in the figure, first, the image input device 7 of the object tracking device 21 obtains one frame of stereo image data from the image pickup device 2 when the image pickup device 2 takes an image of a predetermined area of the road. Then, it is stored in the storage unit 9 (S101).

  Next, the feature point extraction unit 11 extracts the feature point in the stereo shot image using the data of the stereo shot image stored in the storage unit 9 and acquires the three-dimensional information of the extracted feature point. (S102). Next, the correlation window setting unit 13 sets a target correlation window related to the feature points extracted by the feature point extraction unit 11 (S103). The above is the same as the object tracking process shown in FIG.

  Next, the image reduction unit 23 reduces the larger size of the target correlation window image set by the correlation window setting unit 13 and the reference correlation window image stored in the storage unit 9 to the smaller size. (S201). In this case, since the images are collated with the smaller size, the amount of calculation required for the collation processing can be suppressed. Therefore, in this embodiment, an object can be tracked quickly.

  After that, as in the object tracking process shown in FIG. 2, the collation unit 14 collates the image of the target correlation window set by the correlation window setting unit 13 with the image of the reference correlation window stored in the storage unit 9 (S104). The display control unit 10 controls the display device 4 to display the result of the above collation (S105). Then, it returns to step 101 and repeats the said operation | movement.

  By the way, when enlarging an image satisfactorily, a new pixel is added between the pixels, and the pixel value of the additional pixel is interpolated based on the pixel values of neighboring pixels. That is, the information amount (total number of pixel values) of the image after enlargement is larger than the known information amount of the image before enlargement. For this reason, there is a high possibility that the increase information includes erroneous information different from the actual information.

  On the other hand, in the case of reducing an image satisfactorily, a plurality of neighboring pixels are reduced to one pixel. At this time, the pixel values of the remaining pixels are also based on the pixel values of the plurality of neighboring pixels. Will be interpolated. However, since the information amount of the image after the reduction is smaller than the known information amount of the image before the reduction, there is a low possibility that the erroneous information is included.

  Therefore, in this embodiment, it is possible to prevent an increase in erroneous information due to image enlargement, so that the accuracy of matching can be improved. As a result, the object can be tracked with high accuracy. The interpolation method is described in publicly known documents (for example, Mikio Takagi, 1 other person, “New Image Analysis Handbook”, The University of Tokyo Press, September 10, 2004, first edition). In the present application, the description is omitted.

[Embodiment 3]
Next, another embodiment of the present invention will be described with reference to FIGS. For convenience of explanation, the same reference numerals as those in the above embodiment are used for the same elements and processes as those in the above embodiment, and the description thereof is omitted. Further, the object tracking system of this embodiment is different from the object tracking system 1 shown in FIG. 1 only in the object tracking unit of the object tracking apparatus, and the other configurations are the same. Therefore, in the present embodiment, the configuration and processing of the object tracking unit will be mainly described.

  FIG. 11 shows a schematic configuration of the object tracking system 30 according to the present embodiment. As illustrated, the object tracking system 30 includes an imaging device 2, an object tracking device 31, and a display device 4.

  The object tracking device 31 includes an image input unit 7, an object tracking unit 32, a storage unit 9, and a display control unit 10. The object tracking unit 32 uses the stereo moving image data stored in the storage unit 9 and the image data of the reference correlation window to track the feature points of the vehicle in the moving image. The object tracking unit 32 is different from the object tracking unit 8 shown in FIG. 1 in that a search range setting unit (search range setting means) 33 is added between the feature point extraction unit 11 and the correlation window setting unit 13. However, other configurations are the same.

  The search range setting unit 33 sets the search range of the target correlation window in which the collation unit 14 collates the reference correlation window and the image. Specifically, the search range setting unit 33 reads the actual position information of the feature points included in the reference correlation window from the storage unit 9, and determines the range in which the vehicle can move from the actual position of the feature points. Is set as

  Further, the search range setting unit 33 sets the correlation window between the coordinate data in the captured image and the data of the actual position information for only the feature points within the set search range among the feature points extracted by the feature point extraction unit 11. To the unit 13.

  Here, the range in which the vehicle can move refers to the range in which the vehicle can move during a period Δt from the frame of the past captured image corresponding to the reference correlation window to the currently processed frame (latest frame). . This range can be determined using the maximum speed of the vehicle, the lane, and the like.

  In the present embodiment, the range in which the vehicle can move is obtained using the maximum speed of the vehicle. For this reason, the maximum speed vector v of the vehicle is stored in the storage unit 9 in the above-described camera coordinate system.

  In the present embodiment, the search range setting unit 33 uses actual position information of feature points. However, since the camera coordinates and the image coordinates can be easily converted as in the above equation (1), the search range setting unit 33 can also use the position information of the feature points on the image.

  FIG. 12 shows the flow of an object tracking process in the object tracking system 30 having the above configuration. As shown in the figure, first, the image input unit 7 of the object tracking device 31 obtains one frame of stereo image data from the image capturing device 2 when the image capturing device 2 captures a predetermined area of the road. Then, it is stored in the storage unit 9 (S101). The feature point extraction unit 11 extracts the feature points in the stereo shot image using the data of the stereo shot image stored in the storage unit 9, and acquires the three-dimensional information of the extracted feature points (S102). . The above is the same as the object tracking process shown in FIG.

  Next, the search range setting unit 33 sets the search range of the target correlation window in which the collation unit 14 collates the reference correlation window and the image (S301). Specifically, first, the search range setting unit 33 reads the information on the maximum speed vector v of the vehicle from the storage unit 9, and next determines the range (Rx, Ry, Rz) in which the vehicle can move in the camera coordinate system. Calculate based on the formula. Here, Δt indicates a period from the time related to the frame corresponding to the reference correlation window to the time related to the latest frame.

The search range in the camera coordinate system is from the actual position of the feature point included in the reference correlation window to the range (Rx, Ry, Rz) calculated by the above equation (4). FIG. 13 shows the captured image in the latest frame as shown in FIG. 4A and the search range SR set for the reference correlation window RW shown in FIG.

  Next, the search range setting unit 33 excludes feature points outside the search range from the feature points extracted by the feature point extraction unit 11. For the remaining feature points, the correlation window setting unit 13 sets a target correlation window (S103), as in the object tracking process shown in FIG. Then, the collation unit 14 collates the image of the target correlation window set by the correlation window setting unit 13 with the image of the reference correlation window stored in the storage unit 9 (S104), and the display control unit 10 determines the result of the above collation. The display device 4 is controlled to display (S105). Then, it returns to step 101 and repeats the said operation | movement.

  Therefore, the object tracking system 30 according to the present embodiment sets the target correlation window only for the feature points within the search range, and thus can reduce the amount of processing in the correlation window setting unit 13 and the matching unit 14. As a result, the object tracking process can be speeded up.

  In the present embodiment, after the feature point extraction unit 11 performs the feature point extraction process and the three-dimensional information calculation process, the search range setting unit 33 performs the search range setting process. It is not limited.

  For example, the search range setting unit 33 may perform the search range setting process before the feature point extraction unit 11 performs the feature point extraction process. In this case, first, the search range setting unit 33 sets the search range as described above. At this time, it is desirable that the search range setting unit 33 sets the search range in the image coordinate system.

  Next, as shown in FIG. 3, the feature point extraction unit 11 acquires the data of the latest stereo image of the frame from the storage unit 9 (S150), and extracts an edge from the stereo image (S151). Next, the feature point extraction unit 11 calculates three-dimensional information regarding the edges existing in the search range set by the search range setting unit 33 among the extracted edges (S152), and the height is within a predetermined range. A certain edge is extracted as a feature point (S153).

  Therefore, since feature points are extracted only for edges within the search range, the processing amount in the feature point extraction unit 11, the correlation window setting unit 13, and the matching unit 14 can be reduced. As a result, the object tracking process can be speeded up.

  Further, for example, after the correlation window setting unit 13 performs the target correlation window setting process, the search range setting unit 33 may perform the search range setting process. Even in this case, since only the target correlation window including the feature points within the search range is collated with the reference correlation window, the processing amount in the collation unit 14 can be reduced. As a result, the object tracking process can be speeded up.

  When there are a plurality of reference correlation windows, it is necessary to perform matching with the target correlation window for each reference correlation window. Therefore, in the above case, the object tracking device 31 sets a target correlation window for each of all feature points included in the captured image, and then sets a search range for each reference correlation window. It is desirable to collate with the target correlation window.

  Each block provided in the object tracking device 3, 21, 31 according to the present invention, in particular, the object tracking unit 8, 22, 32, and the display control unit 10 may be configured by hardware logic, as follows. You may implement | achieve by software using CPU.

  That is, the object tracking devices 3, 21, 31 have a central processing unit (CPU) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, and a RAM ( random access memory), a storage device (recording medium) such as a memory for storing the program and various data. The object of the present invention is to record the program code (execution format program, intermediate code program, source program) of the control program of the object tracking device 3, 21, 31, which is software that implements the functions described above, so that it can be read by a computer. This can also be achieved by supplying the recorded medium to the object tracking devices 3, 21, and 31 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the object tracking devices 3, 21, and 31 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  In the present invention, since a feature point is extracted from a captured image and an object is tracked by tracking the feature point, a system other than a system that tracks a vehicle traveling on a road and measures the number and speed of vehicles In addition, the present invention is applicable to various devices or systems that require object tracking.

It is a block diagram which shows schematic structure of the object tracking system which is one Embodiment of this invention. It is a flowchart which shows the flow of the object tracking process in the object tracking apparatus of the said object tracking system. It is a flowchart which shows the flow of the extraction process of the feature point in the said object tracking process. FIGS. 4A to 4C are diagrams showing an example of an image acquired in the process of extracting the feature points, and FIG. 4A is a stereo image taken by the imaging device of the object tracking system. One of the photographed images is shown. FIG. 5B shows an image in which an edge is extracted from the photographed image, and FIG. 4C shows a plurality of extracted edges. FIG. 5 shows an image in which only edges that are likely to be automobile edges are left. It is a figure which shows the relationship between the camera coordinate which shows the three-dimensional position of an object on the basis of the said imaging device, and the image coordinate which shows the position on the image of an object. It is a figure which shows the said picked-up image in the previous flame | frame, and the reference correlation window set to this picked-up image. (A) and (b) are diagrams showing the captured image in the latest frame and a target correlation window set in the captured image, and (a) in FIG. The present Example which changes the size of an object correlation window based on the distance to a point is shown, The figure (b) has shown the comparative example whose size of an object correlation window is fixed. It is a figure which shows the case where a template matching is performed with respect to the said picked-up image in the newest flame | frame using the image of the said reference correlation window as a template. It is a block diagram which shows schematic structure of the object tracking system which is another embodiment of this invention. It is a flowchart which shows the flow of the object tracking process in the object tracking apparatus of the said object tracking system. It is a block diagram which shows schematic structure of the object tracking system which is other embodiment of this invention. It is a flowchart which shows the flow of the object tracking process in the object tracking apparatus of the said object tracking system. It is a figure which shows the said picked-up image in the newest flame | frame, and the search range set with respect to the said reference correlation window by the search range setting part of the said object tracking device.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1.20.30 Object tracking system 2 Image pick-up device 3.21,31 Object tracking device 5 1st imaging part 6 2nd imaging part 7 Image input part (image acquisition means)
8.22.32 Object tracking unit (object tracking means)
9 Storage unit (storage means)
11 feature point extraction unit (feature point extraction means)
13 Correlation window setting section (correlation window setting means)
14 Verification part (Verification means)
23 Image reduction part (verification means)
33 Search range setting unit (search range setting means)
RW Reference correlation window TW Target correlation window SR Search range

Claims (11)

An object tracking device comprising: an image acquisition unit that acquires a plurality of captured images created by shooting at different points in time; and an object tracking unit that tracks an object in the captured image based on the acquired plurality of captured images. Because
The image processing apparatus further includes storage means for storing image information of the reference correlation window, with a partial area of the captured image including a feature point characterizing a partial shape of an object in a captured image obtained by past imaging as a reference correlation window,
The image acquisition means acquires a stereo shot image, which is a plurality of shot images shot simultaneously from different positions, as the shot image shot at a certain point in time,
The object tracking means includes
A feature for extracting the feature points using at least two photographed images of the stereo photographed images acquired by the image acquisition means and calculating real position information that is information relating to the actual positions of the extracted feature points. Point extraction means;
The captured image, which is a partial region of the captured image including the feature point extracted by the feature point extracting unit and has a size calculated based on the actual position information of the feature point calculated by the feature point extracting unit Correlation window setting means for setting a partial region of the target area as a target correlation window;
An object tracking apparatus comprising: a collating unit that collates the image of the target correlation window set by the correlation window setting unit and the image of the reference correlation window stored in the storage unit with the same size.   2. The object tracking device according to claim 1, wherein the collating unit collates the image of the target correlation window and the image of the reference correlation window by reducing the larger size to the smaller size. The object tracking means further comprises search range setting means for setting a range in which the feature point can move as a search range based on the position of the feature point included in the reference correlation window,
2. The object according to claim 1, wherein the collating unit collates an image of a target correlation window including a feature point within a search range set by the search range setting unit and an image of the reference correlation window. Tracking device. The actual position information is information of a shooting distance from the shooting device that performs the shooting to the feature point,
The storage means stores the information of the shooting distance related to the feature points included in the reference correlation window in association with the corresponding reference correlation window,
The correlation window setting means has the formula:
(Size of the reference correlation window) = (size of the target correlation window) × (the shooting distance regarding the feature points included in the target correlation window) / (the shooting distance regarding the feature points included in the reference correlation window),
The object tracking device according to claim 1, wherein a size of the reference correlation window is determined based on the reference.   The said collating means memorize | stores in the said memory | storage means as information regarding a new reference correlation window the information regarding the object correlation window with which an image is most similar to the said reference correlation window as a result of the said collation. The object tracking device described.   The object tracking device according to claim 1, wherein the storage unit stores information related to a photographing time point when the photographed image corresponding to the reference correlation window is photographed in association with the reference correlation window.   The object tracking apparatus according to claim 1, wherein the storage unit stores information regarding a plurality of reference correlation windows regarding different feature points. An imaging device including a first imaging unit and a second imaging unit that simultaneously capture images from different positions;
8. The method according to claim 1, wherein a plurality of captured images captured at different points in time are acquired from the imaging device, and an object in the captured image is tracked based on the acquired captured images. An object tracking system comprising: the object tracking device described above.   An object tracking program for operating the object tracking device according to claim 1, wherein the computer functions as the object tracking unit.   A computer-readable recording medium on which the object tracking program according to claim 9 is recorded. An object tracking device that acquires a plurality of captured images created by shooting at different points in time and tracks an object in the captured images based on the acquired captured images. A method for controlling an object tracking device comprising storage means for storing image information of the reference correlation window, using a partial region of the captured image including a feature point characterizing a partial shape of an object as a reference correlation window,
An image acquisition step of acquiring a plurality of captured images created by shooting at different time points;
An object tracking step of tracking an object in the captured image based on the acquired plurality of captured images;
In the image acquisition step, stereo captured images that are a plurality of captured images simultaneously captured from different positions are acquired as the captured images captured at a certain time point,
The object tracking step includes
The feature points are extracted using at least two captured images of the stereo captured images acquired in the image acquisition step, and real position information that is information on the actual positions of the extracted feature points is calculated. A feature point extraction step,
A partial area on the photographed image including the feature points extracted in the feature point extraction step, and a size calculated based on the actual position information of the feature points calculated in the feature point extraction step. A correlation window setting step of setting a partial region of the captured image as a target correlation window;
An object tracking apparatus comprising: a collating step for collating the image of the target correlation window set in the correlation window setting step and the image of the reference correlation window stored in the storage unit with the same size Control method.