JP5106302B2 - Moving object tracking device - Google Patents

Description

  The present invention relates to a moving object tracking device that tracks moving objects in images that are sequentially captured in image processing. In particular, since multiple moving objects frequently intersect in the captured image, such as an interphone with a camera, these intersecting moving objects are extracted as a single change area, and individual movements are extracted. This is suitable for a moving object tracking apparatus in a situation where it is difficult to keep tracking as an object.

  For example, a camera-equipped interphone equipped with a moving object tracking device is installed next to a door of an entrance, and continuously captures a monitoring area in front of the entrance by a camera connected to the moving object tracking device at regular time intervals. When the moving object tracking device tracks the moving object in the image thus captured, it is possible to monitor whether there is a suspicious person such as a stalker in front of the entrance. However, many people, cars, bicycles, etc. come and go, such as when the entrance faces a public place such as a road or parking lot, and multiple moving objects frequently intersect in the captured image. is there. Since these intersecting moving objects are detected as a single change area integrated in the image, the position of each moving object becomes unclear, and then accurately tracked as each moving object There is a problem that it is difficult to continue.

  With respect to such a problem, in Patent Document 1, when a plurality of moving objects intersect while tracking a moving object in a captured image, template matching is performed using a template image of each moving object. Thus, individual positions of a plurality of intersecting moving objects are obtained. In this way, a single integrated change area is divided into areas for each moving object, and by updating the template of each moving object for each divided area, a plurality of moving objects intersect in the screen. It shows how to keep tracking while you are.

JP 2004-013614 A

  However, in the invention disclosed in Patent Document 1, when a plurality of moving objects intersect, the template is updated by using the region divided by template matching as it is, so that a part of the moving objects existing on the back side is on the front side. A template that is hidden by an existing moving object and lacks the relevant part is generated. Therefore, the quality of the updated template is deteriorated as a template for tracking a moving object in the next frame, and the accuracy of image tracking is significantly reduced. In addition, when multiple moving objects intersect, if you try to perform an accurate template update considering that some of the moving objects that exist on the back side are hidden by the moving objects that exist on the front side, Complicated image processing is required, and it becomes difficult to implement in an operating environment where hardware resources are limited.

  For example, it is practically difficult to increase the frame rate in an interphone with a camera that is required to be realized with a simpler hardware configuration than an image monitoring device for intruder detection. For example, if an image can only be acquired and processed for about 2 frames per second, some or all of the frame at the previous time was not shown in the screen for an object with a high moving speed such as a car, Crossing with other moving objects frequently occurs in the frame at the current time. In such a case, since the invention disclosed in Patent Document 1 divides the template based on the entire image of the moving object in the frame at the previous time, appropriate tracking is impossible.

  Accordingly, an object of the present invention is to provide a moving object tracking device capable of accurately tracking an image even when a plurality of moving objects intersect, with the object of solving the above problems. Another object of the present invention is to provide a moving object tracking apparatus with less image processing load.

  In order to solve the above problems, a moving object tracking device according to the present invention includes an image input unit that sequentially acquires images, moving object information that includes image-dependent information and non-image-dependent information used for tracking for each moving object, A tracking unit that includes a storage unit that stores a reference image that does not include a moving object, and a tracking processing unit that sequentially processes an input image and tracks a moving object on the image. The unit includes a change area extraction unit that extracts a change area by performing a difference process between the reference image and the input image, a feature extraction unit that extracts feature information from the change area in the input image, and the feature information and moving object information of the change area An association means for associating the change area with the moving object information using a cross-section, and an intersection determination means for determining whether or not there is an intersection where a plurality of moving object information is associated with one change area by the association means. For each moving object, a moving object information extracting unit that extracts image-dependent image-dependent information and non-image-dependent non-image-dependent information from a changing region in the input image, and a changing region that is determined to have an intersection by an intersection determining unit For the moving object information corresponding to, the non-image dependent information stored in the storage unit is updated using at least the non-image dependent information extracted by the moving object information extracting means, and the intersection determining means determines that there is no intersection. For the moving object information corresponding to the changed area, the image dependent information and the non-image dependent information stored in the storage unit using both the image-dependent information and the non-image-dependent information extracted by the moving object information extraction unit. And moving object information updating means for updating both.

  In the moving object tracking device according to the present invention, the image dependence information in the storage unit includes an image of a region associated with past moving object information in the moving object information as a template. A template matching processing unit that performs a template matching process using a template of the corresponding moving object information with respect to a change area that is determined to have an intersection by the intersection determination unit, and a region that has been matched by the template matching processing unit An area determined to be the foreground among the change areas determined to have intersections, the foreground determination means determining the area at the lowest position as the foreground. For moving object information corresponding to, the image dependence stored in the storage unit using both the extracted image dependence information and non-image dependence information For the moving object information corresponding to the area determined to be not the foreground by updating both the information and the non-image dependent information, only the non-image dependent information stored in the storage unit using the extracted non-image dependent information May be updated.

  In the moving object tracking device according to the present invention, the non-image-dependent information stored in the storage unit includes a region in the moving object information corresponding to the moving object information in the past. The region that has been matched in the matching process, and the region corresponding to the moving object one hour before is in contact with the edge of the image. You may extend within.

  In the moving object tracking device according to the present invention, the associating means is a correspondence table showing the similarity between the moving object one time ago from the moving object information and feature information stored in the storage unit and the change area of the current time. And the change area and the moving object information may be associated with each other so that the sum of the similarities in the correspondence table is maximized.

  According to the present invention, when tracking a plurality of moving objects at the same time in sequentially captured images, only a part of the information about the moving object is updated with respect to the moving object determined to be located behind the other moving objects. I do. As a result, regarding a moving object located behind another moving object, there is a portion that is hidden behind the moving object located in front and is not reflected in the image, so that information that degrades accuracy is not forcibly updated. Therefore, it is possible to prevent a decrease in the accuracy of image tracking due to a deterioration in the quality of information regarding the moving object. For this reason, each moving object can be accurately tracked even in a situation where a plurality of moving objects intersect.

  As a preferred embodiment of the present invention, a moving object tracking device according to the present invention will be described based on an example when mounted on a camera-equipped interphone.

  FIG. 1A is a diagram showing an installation example of an interphone with a camera on which a moving object tracking device according to an embodiment of the present invention is mounted. As shown in the example of FIG. 1A, the intercom with camera 2 is installed on a wall surface 7 such as the side of the entrance door 6. As shown in FIG. 1B, the camera built in the interphone with camera 2 is installed with the optical axis 31 oriented in a substantially horizontal direction, and continuously images the monitoring area in front of the entrance. The moving object tracking device connected to this camera detects suspicious people who stay in the angle of view of the camera for a long time by tracking moving objects in the image while sequentially receiving images taken continuously by the camera. To do.

  FIG. 2 is a block diagram illustrating a configuration of an interphone with a camera in which the moving object tracking device 10 according to the embodiment of the present invention is mounted. As shown in FIG. 2, the moving object tracking device 10 according to an embodiment of the present invention includes an image acquisition unit 110, a tracking processing unit 130, a storage unit 140, a stay determination unit 150, a person determination unit 160, The alarm unit 170 is connected to the imaging device 20.

  The imaging device 20 such as a camera mounted on the camera interphone captures a predetermined monitoring area at a predetermined time interval, and sequentially sends the captured images to the tracking processing unit 130. Hereinafter, the unit of time recorded in the predetermined time is referred to as time. As the imaging device 20, for example, a known device such as an internal camera or an external camera including an imaging element such as a CCD element or a C-MOS element, an optical system component, an A / D converter, or the like is used. it can. Further, the resolution of the imaging device 20 can be selected according to a specific application such as a tracking target, and for example, the NTSC standard, the SDTV standard, or the HDTV standard can be used. Moreover, as a wavelength band used for image capturing, a visible light wavelength or an infrared wavelength is preferably selected according to a specific application such as a tracking target.

  The image acquisition unit 110 is an interface for receiving image signals sequentially sent from the imaging device 20, and receives images from the imaging device 20 via wired or wireless such as SCSI, USB, LAN, dedicated cable, wiring, or the like. The image acquisition unit 110 sends the image received from the imaging device 20 to each unit of the moving object tracking device 10.

  The storage unit 140 can store various programs and various data. For example, a semiconductor memory such as a RAM, a ROM, and an EPROM, a magnetic recording medium such as a hard disk, an optical recording medium such as a CD-ROM and a DVD-R / W, etc. Can be used. The storage unit 140 is connected to the image acquisition unit 110, the tracking processing unit 130, the stay determination unit 150, the person determination unit 160, and the alarm unit 170, and reads and writes various programs and various data according to requests from each unit. . The storage unit 140 stores moving object information 141, a reference image 144, and a suspicious person detection image 145.

  The moving object information 141 is information regarding a past moving object acquired at each unit time stored for each moving object being tracked, and includes image dependency information 142 that depends on an image of the moving object, and an image of the moving object. Non-image-dependent information 143 that does not depend.

  The image dependency information 142 is information extracted based on an input image of a moving object region, for example, including feature information and a template of the moving object. The feature information is information representing the feature of the moving object. In this embodiment, for example, the color histogram or area of the image of the moving object is included, but the average or standard deviation of pixel values, edge information, texture information, and the like are included. Other feature information may be included. The color histogram is feature information representing the color distribution in the object image. For example, the pixel value such as the luminance value of each pixel in the change region is accumulated for each bin of a predetermined width, and each accumulated value is represented by the total number of pixels. Calculated by dividing and normalizing. The template used in the present embodiment is a template image of a moving object detected in the input image one time before, and a template for specifying where the moving object being tracked is located in the input image at the current time. Used in matching. Therefore, any type of template such as various feature vectors and eigenvectors may be used as long as it can be used as a template for template matching.

  The non-image-dependent information 143 includes, for example, the area of the moving object in the input image one hour before, the movement amount of a predetermined number in the past (for example, five times), and the time elapsed since the moving object appeared on the screen ( This is information that does not depend on the input image of the area of the moving object, such as the “existing time”. Here, the area of the moving object is a range that the moving object occupies in the input image, and can generally be represented by its shape, size, and position. In this embodiment, a circumscribed rectangle is used as the area, and the area can be specified by the size and position of the circumscribed rectangle. For example, the position of the center of gravity may be used.

  The reference image 144 is an image that serves as a reference for comparison when extracting the change area. For example, a background image when no moving object is present in the monitoring area is captured in advance and stored in the storage unit 140. Keep it.

  The suspicious person detection image 145 stores an image of a moving object that seems to be a suspicious person when a moving object that has remained in the screen for a certain period of time is detected as a result of tracking the moving object in the image.

  The tracking processing unit 130 processes the images sequentially received from the image acquisition unit 110 to track a moving object in the image. Similar to the stay determination unit 150, the person determination unit 160, and the alarm unit 170, the tracking processing unit 130 reads out a program describing the processing procedure stored in the storage unit 140, and executes a CPU, DSP, MCU, ASIC, ASSP, or the like. It is executed by the processing device. The tracking processing unit 130 is connected to the image acquisition unit 110, the storage unit 140, the stay determination unit 150, the person determination unit 160, and the alarm unit 170.

  As shown in FIG. 2, the tracking processing unit 130 includes a change area extraction unit 131, a feature extraction unit 132, an association unit 133, an intersection determination unit 134, a hierarchical template matching unit 135, and a foreground foreground determination unit. 136, a moving object information extraction unit 137, and a moving object information update unit 138.

  The change area extraction unit 131 includes a reference image 144 that is stored in advance in the storage unit 140 and has no moving object, and an image captured by the imaging device 20 and received via the image acquisition unit 110 (hereinafter referred to as an input image). And a change area in the input image is extracted.

  The feature extraction unit 132 extracts feature information from the input image of the change region for each change region extracted by the change region extraction unit 131 and passes the feature information to the association unit 133.

  The associating unit 133 extracts each change extracted by the change region extracting unit 131 based on the feature information of each changed region extracted by the feature extracting unit 132 and the moving object information 141 of each moving object stored in the storage unit 140. The area and the most likely moving object existing in the change area are associated with each other.

  The intersection determination unit 134 determines whether or not the moving object that has not been associated with any change area by the association unit 133 is associated with the change area that is already associated with another moving object. It is determined whether or not a plurality of moving objects intersect in the region.

  As a result of the determination by the intersection determination unit 134, the hierarchical template matching unit 135 detects the region of each moving object by template matching using a template image in a change region associated with a plurality of intersecting moving objects. Details of the hierarchical template matching unit 135 will be described later.

  Whether the foreground determination unit 136 is a moving object located in the foreground among a plurality of moving objects intersecting in each change region based on the position of the moving object region detected by the hierarchical template matching unit 135 Or a moving object located behind another moving object.

  The moving object information extraction unit 137 extracts non-image-dependent information 143 that does not depend on the image from the information about the moving object region detected by the hierarchical template matching unit 135, and depends on the image from the input image of the moving object region. The image dependence information 142 to be extracted is extracted.

  The moving object information updating unit 138 stores the image dependent information 142 and the non-image dependent information 143 extracted by the moving object information extracting unit 137 according to whether or not the moving object intersects with another moving object. The moving object information 141 of the moving object stored in the unit 140 is updated. When the intersection determination unit 134 determines that there is only one moving object in the change area, the moving object information update unit 138 updates all the moving object information 141. Further, when it is determined by the intersection determination unit 134 that a plurality of moving objects intersect, if the moving object is determined to be located in the foreground by the foreground determination unit 136, all the moving object information 141 is updated. However, if it is determined that the moving object is behind another moving object, only the non-image-dependent information 143 in the moving object information 141 is updated.

  The stay determination unit 150 checks the existence time stored in the storage unit 140 for each moving object, and if it exists in the image longer than a predetermined time (for example, one minute), it determines whether it is a human. Information on the area of the moving object is sent to the person determination unit 160 for examination.

  The person determination unit 160 determines whether or not the moving object that has been present in the screen for a longer time than the predetermined time sent from the stay determination unit 150 is a human, and the determination that the person is a human is an alarm. The result is sent to the unit 170. Further, the image at that time may be stored in the storage unit 140 as the suspicious person detection image 145.

  When the alarm unit 170 receives the result from the person determination unit 160, the alarm unit 170 notifies the user that a suspicious person has been detected by an alarm lamp such as an LED or an alarm buzzer. A message stored in advance may be played as a voice through a speaker or displayed on a display. The suspicious person detection image 145 stored in the storage unit 140 can also be output via an output device (not shown) such as a display or a printer.

[Operation of the Invention]
FIG. 7 is a flowchart showing an operation flow of the moving object tracking device 10 according to the embodiment of the present invention. The details of the operation of each part of the moving object tracking device 10 will be described with reference to FIG.

  In step S <b> 100, the moving object tracking device 10 stores a background image captured in advance in the situation where the moving object does not exist within the angle of view of the imaging device 20 by the imaging device 20 as the reference image 144 in the storage unit 140. , Initialization for moving object tracking processing such as setting the moving object information 141 regarding the moving object to be tracked to an initial state is performed.

  In step S <b> 200, the moving object tracking device 10 receives images captured by the imaging device 20 at predetermined time intervals via the image acquisition unit 110 and passes them to the tracking processing unit 130. In this way, the moving object tracking device 10 repeats the processing of steps S300 to S1400 by the tracking processing unit 130 each time input images are sequentially sent from the imaging device 20 at regular time intervals.

  In step S300, the tracking processing unit 130 performs a difference process between the input image newly received from the imaging device 20 and the reference image 144 stored in the storage unit 140 by the change area extraction unit 131, thereby Extract change areas.

  The change area extraction unit 131 compares the input image captured by the imaging apparatus 20 and received via the image acquisition unit 110 with the reference image 144 acquired in advance and stored in the storage unit 140, and a region having a difference is detected. Is extracted as a change area, and the result is passed to the feature extraction means 132. For example, the change area extraction unit 131 performs difference processing on the pixel values of the corresponding pixels of the input image and the reference image 144, and executes a method such as binarization and labeling on a pixel group having a certain difference value or more. . In this way, a change area different from the reference image 144 in the input image, that is, an area where a moving object is estimated to exist is extracted. In addition, about the method of calculating | requiring the difference between pixel values, since well-known various methods can be used, description about a detail is abbreviate | omitted.

  In step S400, the tracking processing unit 130 performs image processing on the input image of the change area extracted by the feature extraction unit 132, and calculates feature information.

  The feature extraction unit 132 divides the change region extracted by the change region extraction unit 131 into groups for each block, attaches a label to each group (hereinafter referred to as a label), and inputs an input image of the region for each label. Feature information is extracted from.

  In step S <b> 500, the tracking processing unit 130 is based on the moving object information 141 of the moving object stored in the storage unit 140 and the feature information of the changed area extracted by the feature extracting unit 132 by the association unit 133. The moving object and the change area are associated with each other.

  The associating unit 133 is configured so that the moving object information 141 regarding the moving object to be tracked detected from the input image one hour before stored in the storage unit 140 and the feature extracting unit 132 from each change area of the input image at the current time. Based on the similarity calculated from the extracted feature information, it is associated with which change area of the input image at the current time each moving object detected one hour ago is most likely to exist.

  FIG. 3 is a diagram illustrating an example of a one-to-one correspondence between a moving object in the input image one hour before and a change area in the input image at the current time. An example of association by the association unit 133 in this embodiment will be described with reference to the drawings.

  FIG. 3 shows an example in which the vehicle 801 and the person 802 that are separated from each other in the input image one time before in FIG. 3A intersect in the input image at the current time in FIG. As shown in FIG. 3C, change areas 805, 806, and 807 are extracted from the input image at the previous time, and identifiers ID1, ID2, and ID3 are assigned to the change areas as shown in FIG. Give. On the other hand, as shown in FIG. 3D, in the input image at the current time, the vehicle 801 and the person 802 are extracted as one change area 808 integrated and intersect, and the person 803 is a single person. As the change area 809, the vehicle 804 is extracted as a single change area 810. As shown in FIG. 3F, identifiers IDA, IDB, and IDC are assigned to these extracted change areas, respectively. In this embodiment, the region is specified by setting the X axis in the horizontal direction and the Y axis in the vertical direction with the upper left corner of the image as the origin. In the example of FIG. 3, the association unit 133 associates the moving objects 802 and 803 and the new moving object 804 with the highest “correspondence” one time ago with respect to the individual change areas IDA, IDB, and IDC, respectively. And a moving object existing in the change area. A moving object 801 one hour before that is not associated with any change region is temporarily determined to be in a lost state (untrackable state).

ここで、各類似度の重み付けｗ_p、ｗ_h、ｗ_aは、３種類の類似度の総和が１となるように予め定めた正数である。尚、各類似度の算出方法としては、多様な周知の方法を使用することができるので詳細な説明は省略する。 Here, the “correspondence degree” is an index indicating how much the characteristics of the moving object one hour ago and the change area of the current time match, and the moving object information 141 of the moving object one hour ago and the current time It is obtained based on the similarity between the change area and the feature information. In this embodiment, in calculating the correspondence, the degree of overlap between the predicted position of the moving object and the change area of the current time (position similarity), the color histogram of the moving object at the previous time, and the color histogram of the change area of the current time The similarity (color histogram similarity) and the ratio (area similarity) of the area of the moving object at the previous time and the area of the change area at the current time are 1.0, respectively. In some cases, it is a positive number of 1.0 or less, and when there is no common point, it is obtained by normalization so that it is 0, and it is calculated as follows, but it is not limited to this, and various known methods should be used. Can do.

Here, the weights w _p , w _h , and w _a of the similarities are positive numbers determined in advance so that the sum of the three types of similarities is 1. In addition, since various well-known methods can be used as a calculation method of each similarity, detailed description is abbreviate | omitted.

  The predicted position of the moving object is the predicted moving speed predicted from the area of the moving object one time before that is the non-image dependent information 143 stored in the storage unit 140 and the past movement amount that is also the non-image dependent information 143. And calculated based on In this embodiment, it is assumed that each moving object is moving at a constant speed between the previous time and the current time, and the position obtained by adding the predicted moving speed to the position of the moving object area at the previous time is the current time. Is used as the predicted position of the moving object. The predicted moving speed can be predicted using a variety of prediction methods such as a prediction method using a first-order AR model and a Kalman filter, based on past movement amounts stored in the storage unit 140.

  In the present embodiment, the method of calculating the correspondence using the predicted position of the moving object, the position of the change area of the current time, the color histogram, and the area is illustrated, but the correspondence is defined using other feature information. May be. In addition, it can be calculated by various methods such as arithmetic mean and geometric mean during synthesis.

The associating unit 133 determines the degree of new appearance and the degree of lost according to the following procedure when the degree of correspondence between the moving object at the previous time and the change area at the current time is obtained.
(I) A correspondence table in which correspondences are arranged in a matrix with the identifier of each moving object at the previous time associated with each row and the identifier of each change region at the current time associated with each column is created.
(Ii) A maximum value of correspondence is obtained for each column of the correspondence table corresponding to each change area at the current time, and a value obtained by subtracting the maximum value from 1.0 is set as the degree of new appearance of the moving object in the change area, A line indicating the degree of new appearance for each change area is added to the correspondence table.
(Iii) The maximum value of the correspondence is obtained for each row of the correspondence table corresponding to each moving object at the previous time, and the moving object disappears from the input image at the current time by subtracting the maximum value from 1.0. As the degree of lost, a column indicating the degree of lost for each moving object is added to the correspondence table.

  FIG. 3 (h) shows an example of the correspondence table created in this way. Correspondence means 133 always selects only one correspondence level per column with exception of the column representing lost, and one correspondence per row with exception of the row representing new appearance, from the correspondence table. The moving object at the previous time is associated with the change area at the current time so that the sum of the degrees becomes the maximum. At this time, a plurality of correspondence levels may be selected from one row representing a new appearance and one column representing a lost. That is, some of the tracking target moving objects are associated with some of the change areas on a one-to-one basis, and the moving objects that are not associated with the change areas are in a lost state and are associated with the moving objects. It is determined that a new moving object has appeared in the change area that did not exist. For example, an optimization method such as the Hungarian method can be used to obtain a combination of associations that maximizes the sum of correspondences based on the correspondence table. Details of the Hungarian law are described in, for example, Hiroshi Sadamichi, “Management Science”, Ohmsha, 1989.

  In the correspondence degree table of FIG. 3H, correspondence degrees surrounded by circles are selected. Therefore, in the example of FIG. 3, the change areas IDA and ID2 (human), the change areas IDB and ID3 (human) are associated, and ID1 (vehicle) is once determined to be lost (however, the intersection determination means 134 described later determines that). Judgment is corrected). In the change area IDC, it is determined that a new moving object has appeared.

  In step S <b> 600, the tracking processing unit 130 may exist in a change area integrated with the other moving objects among the moving objects determined to be lost by the association unit 133 by the intersection determination unit 134. A moving object is associated with the change area.

  The intersection determination unit 134 checks whether or not the circumscribed rectangle of the predicted position of each moving object determined to be lost by the association unit 133 overlaps the circumscribed rectangle of each change area at the current time. If there is a circumscribed rectangle of the change area that overlaps the circumscribed rectangle of the moving object that has been determined to be lost, the intersection determination unit 134 determines that the moving object is an integrated change area in a state where it intersects with another moving object. It is determined that it has been extracted. Then, the moving object is associated as a moving object existing in the change area. Thus, the intersection determination unit 134 establishes a correspondence relationship between the moving object determined to be lost and the change area. When one moving object overlaps a plurality of change areas, the intersection determination unit 134 associates the moving object with the change area having the largest overlap.

  The processing of the intersection determination unit 134 will be described with reference to FIG. A circumscribed rectangular area indicated by a broken line in FIG. 3F represents a predicted position at the current time of the moving objects ID1, ID2, and ID3 at the previous time. For the sake of easy understanding, FIG. 3 (g) shows only the rectangle information of FIG. 3 (f) extracted. Here, rectangles RA, RB, and RC indicated by solid lines represent circumscribed rectangles IDA, IDB, and IDC of the change area at the current time, respectively, and rectangles R1, R2, and R3 indicated by broken lines respectively represent moving object ID1. , The circumscribed rectangle of the predicted position at the current time of ID2 and ID3. In the example of FIG. 3, the intersection determination unit 134 determines that the moving object ID1 is once lost, but the circumscribed rectangle R1 of the predicted position is already circumscribed rectangle RA of the change area IDA associated with the moving object ID2. Therefore, the moving object ID1 is also associated with IDA together with the moving object ID2.

  In step S <b> 700, the tracking processing unit 130 regards each change region determined to have a plurality of intersecting moving objects by the intersection determination unit 134 (Yes in step S <b> 750), and exists in the change region by the hierarchical template matching unit 135. The area of the moving object to be identified is specified.

  In step S800, the hierarchical template matching unit 135 extracts a plurality of intersecting moving object regions determined by the intersection determining unit 134 to exist in each change region by template matching for each moving object, and displays the extracted regions. The area of the moving object in the input image at the time is corrected, and if necessary, the extracted area is corrected. Details of the hierarchical template matching unit 135 will be described later.

  In step S900, the tracking processing unit 130 causes the moving object information extracting unit 137 to extract moving object information based on the moving object region specified by the hierarchical template matching unit 135 and the input image of the region.

  The moving object information extracting unit 137 extracts non-image-dependent information such as the moving object region and the moving amount based on the moving object region specified by the hierarchical template matching unit 135. The moving object information extraction unit 137 extracts image-dependent information including feature information such as a color histogram and an area of the moving object image from the input image of the moving object region. Note that when the moving object is located behind another intersecting moving object in the change region, the accuracy of the image-dependent information may be deteriorated, so the image-dependent information may not be extracted.

  In step S <b> 1000, the tracking processing unit 130 updates the moving object information 141 based on the moving object information extracted by the moving object information extracting unit 137 by the moving object information updating unit 138.

  The moving object information update unit 138 determines whether or not the moving object intersects with another moving object based on the processing result of the intersection determination unit 134, and when the other moving object intersects, the foreground The determination unit 136 determines whether or not the moving object is located in the foreground, and updates all or part of the moving object information 141 of the moving object stored in the storage unit 140 according to the result.

  When the plurality of moving objects intersect in the change area, the foreground determination unit 136 sets the moving object in which the lower end of the area of each moving object extracted by the hierarchical template matching unit 135 is located at the bottom as the foreground. It is determined that it is located.

  With reference to FIG. 1 and FIG. 5, a method for determining the front-rear relationship of a moving object in the present embodiment will be described. As shown in FIG. 1A, in this embodiment, it is assumed that the moving object tracking device according to the present invention is applied to an interphone with a camera beside the entrance. The intercom with camera is usually installed at a height that can be reached by a human hand, and the mounted camera is oriented in a direction substantially parallel to the ground, that is, in a substantially horizontal direction.

  In the example shown in FIG. 1B, the human 3 and the vehicle 4 that are moving objects are lined up and down as viewed from the camera on the ground surface 30 that is substantially parallel to the optical axis direction 31 photographed by the camera. The human 3 is closer to the camera than the vehicle 4. Therefore, in the image photographed by the camera, the human 3 appears in the foreground and the vehicle 4 appears in the back of the human 3. A part of the vehicle 4 intersects with the human 3 and is reflected behind it, so that it is hidden behind the human 3 and cannot be seen. In this case, the ratio of the distance 36 between the lower end of the person located in front of the camera and the lower end of the imageable range to the entire imageable range 34 is the lower end of the vehicle located on the rear side. Since the distance 37 between the lower end of the shootable range is smaller than the ratio of the entire shootable range 35, the lower end of the person in front is positioned below the lower end of the vehicle on the rear side in the taken image. To do. Therefore, in many cases as well as in the present embodiment, this method can be applied to determine the context of the photographed object.

  FIG. 5 is a diagram illustrating the context determination process in the moving object tracking device 10 according to the embodiment of the present invention. Here, an example in which the front / rear determination is performed in the case of FIG. 3B is shown. FIG. 5A shows the changed area 1001 extracted by the changed area extracting unit 131, and FIG. 5B is extracted as a result of performing template matching using the template images T1 and T2 of FIG. Represents circumscribed rectangles 1002 and 1003 of the region. Since the lower end of the rectangle 1003 of the human matching position is located below the lower end of the rectangle 1002 of the vehicle matching position, it is determined that the human is positioned in front of the vehicle. In this way, a portion where the template image T2 arranged at the matching position and the change region overlap is extracted as a human region at the current time. The position of the remaining vehicle is determined by performing template matching again on an area 1004 (FIG. 5C) obtained by removing a human area from the original change area diagram 1001 of FIG.

  The moving object information updating unit 138 updates all the moving object information 141 when there is only one moving object in the change area. When the moving object intersects with another moving object in the change area, if the moving object is positioned in the foreground, all of the moving object information 141 is updated. However, when the moving object is located behind another moving object, only the non-image dependent information 143 in the moving object information 141 is updated, and the image dependent information 142 is not updated.

  As described above, the moving object information updating unit 138 performs, for example, feature information such as a color histogram and an area, or a template image for a moving object that intersects with another moving object in the change region and is located behind the other moving object. The image dependence information 142 that depends on the image data such as is not intentionally updated. With respect to the image-dependent information 142, the value of the previous time is directly used and used in the tracking process of the moving object in the input image at the next time. By doing so, a part of the image of the moving object behind is hidden by the previous moving object. Therefore, if the image dependent information 142 is updated based on the image of the entire area of the moving object behind, the accuracy deteriorates. To prevent it. In this way, in order to improve the accuracy of such information update, it is possible to track moving objects without complicated and advanced processing such as analyzing the area hidden by the moving object in front and increasing the frame rate. Information with sufficient accuracy can be maintained.

  However, the moving object information updating unit 138 updates the moving object behind the non-image dependent information 143 that does not depend on the image data such as the area of the moving object and the moving amount. This is because, when a part of the moving object is hidden behind the moving object in the front, updating the information based on the entire area of the moving object behind does not greatly affect the accuracy of the information. Such non-image-dependent information 143 is useful as a determination criterion in the tracking process of the moving object in the input image at the next time.

  On the other hand, when there is only one moving object in the change region, or when the moving object information updating unit 138 intersects with another moving object but is located in the foreground, not only the non-image-dependent information 143 but also the color histogram In addition, the image dependency information 142 including feature information such as an area and a template image is also updated. In this case, part of the moving object is not hidden behind other moving objects, and the entire image of the entire area of the moving object is obtained, so that the correct image-dependent information 142 is extracted.

  FIG. 4 is a diagram illustrating an example of updating a template image in the moving object tracking device 10 according to the embodiment of the present invention. The update of the template image 142 when moving objects intersect will be described with reference to FIGS. 3 and 4.

  As shown in FIG. 3A, when the vehicle 801 and the question 802 are separated and appear in the image, the respective template images are T1 and T2 in FIG. 4A. At the next time, as shown in FIG. 3B, the vehicle 801 and the person 802 intersect in the image. In this case, the moving object information updating unit 138 indicates the template image of the person 802 as T2 ′ in FIG. 4B because the person 802 intersects the vehicle 801 of another moving object but is positioned in the foreground. Update as follows. On the other hand, since part of the vehicle 801 is hidden behind the human 802 and cannot be seen in its entirety, the moving object information updating unit 138 does not update the template image of the vehicle 801 and uses the template image T1 of the previous time as it is. Use. Therefore, deterioration of the accuracy of the template image of the vehicle 801 can be prevented.

  In step S1005, the tracking processing unit 130 increments the tracking success count stored in the storage unit 140 by one for the moving object that has been successfully tracked. On the other hand, if the tracking fails and is lost, the tracking success count is reset to zero. When it is determined that the moving object has moved out of the screen, the tracking processing unit 130 deletes the moving object information 141 regarding the moving object from the storage unit 140.

  In step S1100, the moving object tracking device 10 determines whether or not the moving object successfully tracked by the tracking processing unit 130 exists in the image for longer than a predetermined time by the stay determination unit 150.

  For example, the stay determination unit 150 determines whether or not the moving object has been present in the image longer than a predetermined time by determining whether or not the tracking success number of the moving object is greater than a predetermined number. Determine whether. In addition, the stay determination unit 150 updates the existence time for the moving object present in the input image acquired at the current time, stores the existence time in the storage unit 140, and the moving object exists for a long time. It may be used to determine whether or not it exists.

  If the moving object is present in the image longer than a predetermined time (Yes in step S1100), in step S1200, the moving object tracking device 10 determines whether the moving object is a person by the person determination unit 160. Determine whether or not.

  The person determination unit 160 determines whether or not the moving object determined to be present in the screen for a longer time than the predetermined time by the stay determination unit 150 is a human being. The result is output to (S1300). Then, the moving object image is stored in the storage unit 140 as the suspicious person detection image 145. Further, for example, other information such as a time period during which the suspicious person stays may be stored in the storage unit 140 together. Further, for example, when a user requests display of such information via an input device such as a switch button or a touch panel in a base unit of an interphone installed indoors, the storage unit 140 is accessed via an output device such as a display. The stored information may be displayed.

  Various known methods can be used as a method for determining whether or not the moving object is a human in the human determination unit 160. For example, P. Viola, M. Jones & D. Snow, “Detecting Pedestrians Using Patterns of Motion and Appearance”, IEEE International Conference on Computer Vision, pp.734, 0ct, As described in 2003, it can be realized by a cascade classifier based on Haar-Like features learned by AdaBoost using human images of various appearances taken in advance.

  When it is determined that the moving object is a human (in the case of Yes in step S1200), in step S1300, the moving object tracking device 10 warns the user with light, sound, or the like by the alarm unit 170, and stores it in the storage unit 140. The stored suspicious person detection image 145 is displayed on an external device, for example, a monitor of an intercom base unit installed indoors.

  FIG. 8 is a flowchart showing an operation flow of the hierarchical template matching unit 135 in the moving object tracking device 10 according to the embodiment of the present invention. Details of the operation of the hierarchical template matching unit 135 will be described with reference to FIG.

  The hierarchical template matching unit 135 repeats the processes of steps S801 to S806 for each moving object that is associated with the change area to be processed.

  In step S <b> 801, the hierarchical template matching unit 135 performs template matching for each moving object associated with the change area to be processed by the intersection determination unit 134 using the template image, and most closely matches in the change area. The position to be extracted is extracted. As a template matching index, various indices such as a normalized correlation with respect to a pixel value between corresponding pixels of the change area and the template image, a sum of absolute differences, and a sum of squared differences can be used.

  In step S803, the hierarchical template matching unit 135 determines the context based on the position of each moving object extracted by template matching, and identifies the moving object located in the foreground. In the present embodiment, like the foreground determination unit 136 described above, it is determined that the moving object whose lower end is located on the lower side is located on the front side. Here, the moving object determined to be the foreground in the first loop of steps S801 to S806 may be stored in the storage unit 140 and used for determination by the foreground determination unit 136.

  In step S805, the hierarchical template matching unit 135 sets a portion where the template image of the moving object located in the foreground arranged at the matching position and the change area overlap as a new area in the input image of the moving object at the current time. Remove the region from the change region.

  The hierarchical template matching unit 135 repeats a series of processes from step S801 to step S806 for the remaining moving objects that are not located in the foreground, and repeats the number of moving objects that exist in the change area in order from the moving object in the foreground. Thus, the area of each moving object is extracted.

  In step S807, the hierarchical template matching unit 135 identifies the areas of all the moving objects existing in the change area, and then determines whether the ratio of the remaining change areas that did not correspond to any moving objects is equal to or greater than a predetermined ratio. Determine whether or not. If the ratio is equal to or less than the predetermined ratio (No in step S807), the remaining area is associated with the moving object that is closest in distance.

  On the other hand, if a change area of a predetermined ratio or more remains (Yes in step S807), does the hierarchical template matching unit 135 include any moving objects at the previous time that are in contact with the screen edge? It is verified whether or not (step S808). If there is no moving object in contact with the screen edge at the previous time (No in step S808), the hierarchical template matching unit 135 associates all remaining change areas as newly appearing moving objects (step S811).

  In step S808, the hierarchical template matching unit 135 identifies a moving object whose shape is likely to change over time from the change area, and corrects the area of the identified moving object. For example, the hierarchical template matching unit 135 determines whether there is a moving object corresponding to the change area that is in contact with the screen edge in the acquired image at the previous time. If there is something that touches the screen edge (Yes in step S808), the hierarchical template matching unit 135 expands the matching result of the moving object in the change area in a direction parallel to the moving direction (step). S809). When a plurality of moving objects are in contact with the screen edge at the previous time, the area is expanded in order from the front object. In this way, the hierarchical template matching unit 135 can associate the area of the moving object that has not been associated because the entire moving object was not reflected in the screen at the previous time from the remaining change area.

  In this way, if a moving object that touches the screen edge at the previous time is moving in the direction of entering the screen, it is considered that the current time has taken the part that was off the screen at the previous time. Therefore, it is considered that the portion that was not visible at the previous time continues in the direction opposite to the moving direction with respect to the portion that was visible. In this way, the hierarchical template matching unit 135 corrects the region by expanding the region after template matching of the moving object in the direction opposite to the moving direction. In the present embodiment, it is assumed that the imaging device 20 is installed so as to face in a substantially horizontal direction like a camera interphone. Therefore, there is no problem even if the moving direction is limited to the horizontal direction.

  FIG. 6 is a diagram illustrating an example of region correction when there is a moving object in contact with the screen edge in the input image at the previous time. With reference to FIG. 6, a description will be given of correction processing of a moving object region when a moving object that is in contact with the screen edge at the previous time intersects with another moving object at the current time.

  As shown in FIG. 6A, at the previous time, the image of the vehicle 901 is in contact with the left end of the screen and only a part is seen, and the human 902 can see the whole body, and as shown in FIG. Let us consider an example in which the vehicle 901 and the person 902 cross and overlap each other. The change areas at the previous time are as shown in FIG. 6B, and are given identifiers ID4 and ID5, respectively. Further, as shown in FIG. 6D, at the current time, the vehicle 901 and the person 902 are integrated by the intersection and extracted as one change area ID6. Therefore, it is necessary to specify the area of each moving object by template matching and extract the specified area from the change area. However, since the moving object is not in an intersecting state at the previous time, the template images of the vehicle 901 and the person 902 in FIG. 6A are as shown at T4 in FIG. 6E and T5 in FIG. 6F, respectively. It is. Thus, only a part of the vehicle 901 is shown in the input image at the previous time, and T4 in FIG. 6E is stored as a template image of the vehicle 901. Therefore, as a result of template matching using the template image T4, as shown in FIG. 6G, the person 902 is associated with ID5 ', the vehicle 901 is associated with ID4', and ID6 'remains. However, since the vehicle is in contact with the left edge of the screen at the previous time and is moving in the right direction of the screen, the vehicle area ID4 ′ is expanded in the left-right direction, so that ID6 ′ is finally included in the vehicle area. .

  After the above processing, if the change area still remains more than a predetermined threshold (Yes in step S810), the hierarchical template matching unit 135 uses the remaining area in step S811 as a newly appearing moving object. Correspond as a region of.

  With respect to the newly appearing moving object, the hierarchical template matching unit 135 extracts the moving object information of the new moving object based on the specified area and the input image of the area, and stores the extracted moving object information in the storage unit 140. When the remaining change area is equal to or less than a predetermined ratio, the hierarchical template matching unit 135 associates the remaining change area with the moving object that is closest in distance (for example, Mahalanobis distance).

  According to the present invention, when tracking a plurality of moving objects in an image, the front-rear position with respect to a camera that captures a plurality of intersecting moving objects is determined. Don't dare to update image-dependent information. Therefore, when a plurality of moving objects intersect in the image, the template image of the moving object located on the rear side is forcibly updated even though there is a portion that is hidden behind the moving object located on the front side. Therefore, it is possible to prevent a decrease in image tracking accuracy due to the deterioration of the quality as a template. For this reason, each moving object can be accurately tracked even in a situation where a plurality of moving objects frequently intersect. Therefore, for example, even when applied to an operating environment in which hardware resources such as an interphone with a camera are limited, it is possible to accurately detect a suspicious person (such as a stalker) who stays in front of the interphone for a long time.

  Also, based on the positional relationship in the input image, that is, by determining that the lower end of the moving object is positioned lower in the input image, the determination of the front-rear relationship of the moving object is easily performed. In particular, by applying the present invention to an interphone with a camera that has to point the camera in a substantially horizontal direction, it is possible to accurately detect a suspicious person (stalker) who stays in front of the entrance for a long time. Furthermore, even if a moving object that was not partially or wholly displayed on the screen in the frame acquired at the previous time intersects with another moving object in the current time frame, the area is By updating, more accurate image tracking is performed. Further, by associating a moving object most likely to be located in the foreground with each change area in the current input image, processing such as intersection determination is facilitated.

  As described above, non-image-dependent information that is updated for all moving objects and image-dependent information that is not updated for moving objects located behind other moving objects are limited to this embodiment. There is no need to For moving objects located behind other moving objects, considering the update accuracy according to the characteristics of the moving object to be tracked, the performance of the imaging device to shoot, the situation of the shooting location, the operating environment of image processing, etc. Information that will not be updated can be selected appropriately. There is a case where information about a moving object located behind another moving object is not updated at all.

  In this embodiment, in order to explain the advantage that each moving object that intersects can be accurately tracked even in an environment where hardware resources such as an interphone with a camera of the present invention are limited, an example of an interphone with a camera will be described. However, the present invention is not limited to this example. It is also effective in an environment having other limited hardware resources. Further, the present invention may provide the same effect even in a high-performance image processing environment having abundant hardware resources depending on the characteristics of the moving object to be tracked, the image capturing conditions, and other situations.

It is a figure which shows the example of installation of the intercom with a camera by which the moving object tracking device which is one Embodiment of this invention was mounted. It is a block diagram which shows the structure of the intercom with a camera by which the moving object tracking device which is one Embodiment of this invention was mounted. It is a figure which shows the example of 1 to 1 matching with the moving object in the input image of 1 time ago, and the change area | region in the input image of the present time. It is a figure which shows the example of the update of the template image in the moving object tracking device which is one Embodiment of this invention. It is a figure which shows the context determination process in the moving object tracking device which is one Embodiment of this invention. It is a figure which shows the example of area | region correction when the moving object which touches the screen edge exists in the input image of the previous time. It is a flowchart which shows the operation | movement flow of the moving object tracking apparatus which is one Embodiment of this invention. It is a flowchart which shows the operation | movement flow of the hierarchical template matching in the moving object tracking device which is one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Moving object tracking apparatus 20 Imaging apparatus 110 Image acquisition part 130 Tracking process part 131 Change area extraction means 132 Feature extraction means 133 Association means 134 Intersection determination means 135 Hierarchical template matching means 136 Foreground foreground determination means 137 Moving object information extraction means 138 Moving object information update means 140 Storage unit 141 Moving object information 142 Image dependent information 143 Non-image dependent information 144 Reference image 145 Suspicious person detection image 150 Stay determination unit 160 Person determination unit 170 Alarm unit

Claims (4)

An image input unit that sequentially acquires images; a moving unit information that includes image-dependent information and non-image-dependent information used for tracking for each moving object; and a storage unit that stores a reference image that does not include the moving object; A moving object tracking device comprising a tracking processing unit that sequentially processes the input image and tracks a moving object on the image,
The tracking processing unit
A change area extraction means for extracting a change area by performing a difference process between the reference image and the input image;
Feature extraction means for extracting feature information from the change region in the input image;
Association means for associating the change area with the moving object information using the feature information of the change area and the moving object information;
Crossing determination means for determining the presence or absence of a crossing in which a plurality of moving object information is associated with one change area in the association means;
For each moving object, moving object information extracting means for extracting image-dependent image-dependent information and non-image-dependent non-image-dependent information from a change area in the input image;
Among the plurality of moving object information corresponding to the change area determined to have an intersection by the intersection determination unit, the moving object information determined to be located in the foreground is the image dependency extracted by the moving object information extraction unit. Updating both information and non-image dependent information, and for the moving object information determined not to be located in the foreground, only the non-image dependent information extracted by the moving object information extracting means,
Moving said the moving object information corresponding to the change area where it is determined that no intersect at intersection determining means for updating both the image-dependent information and the non-image-dependent information extracted by the moving object information extraction means Object information updating means;
A moving object tracking device comprising: The image-dependent information in the storage unit includes an image of a region associated with past moving object information among the moving object information as a template,
Furthermore,
The tracking processing unit performs template matching processing using the template of the corresponding moving object information for the change area determined to be crossed by the cross determination unit;
Moving object tracking according to regions in the lowest position in the region matching bets were in the template matching process unit in claim 1 that immediately Bei and a top foreground judging means judges that positioned in the outermost foreground apparatus. The non-image-dependent information stored in the storage unit includes a region in the moving object information associated with the moving object information in the past,
The moving object information extraction unit is an area that has been matched in the template matching process, and the area that has been matched with respect to an area in which an area corresponding to the moving object one hour before is in contact with an image end The moving object tracking device according to claim 2, wherein the moving object tracking device is expanded within the range of the change region in parallel with the moving direction. The association unit generates a correspondence table indicating a correspondence degree based on the similarity between the moving object one hour before and the change area of the current time from the moving object information and the feature information stored in the storage unit. 4. The moving object tracking device according to claim 1, wherein the change area and the moving object information are associated with each other so that the total sum of the correspondences in the correspondence degree table is maximized.

Images