JP2010087882A - Moving object tracking device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving object tracking device that not only precisely tracks an image but also has small processing load thereof even if water drop is deposited to an image capturing apparatus. <P>SOLUTION: The moving object tracking device 10 includes: a feature extracting means 1313 for extracting feature information from a magnification change region where a change region of a reference image is magnified and an input image change region; a water drop determining means 1314 for determining that the change region is a water change region formed by the water drop if these feature information is identical each other and that the change region is a non-water drop change region formed by moving objects other than the water drop if not identical; a matching means for matching the non-water drop change region to moving object information 141 by using the feature information and moving object information 141 extracted from the non-water drop change region; and a moving object information updating means 134 for updating the moving object information 141 matched to the non-water drop change region based on the matching of the non-water drop region to the moving object information 141. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a moving object tracking device that tracks each of a plurality of moving objects in images sequentially captured in image processing. In particular, the present invention relates to a moving object tracking device capable of continuing tracking of a moving object even in a situation where water droplets adhere to a cover or the like of an imaging device.

  For example, a camera-equipped interphone equipped with a moving object tracking device is installed next to a door of an entrance, and continuously captures a monitoring area in front of the entrance by a camera connected to the moving object tracking device at regular time intervals. When the moving object tracking device tracks the moving object in the image thus captured, it is possible to monitor whether there is a suspicious person such as a stalker in front of the entrance. However, for example, when a camera is installed outdoors, such as an interphone with a camera, water droplets may frequently adhere to the cover of the imaging device due to rain or the like. At this time, unless the change area changed compared to the background image is due to the movement of the object or due to the attachment of water droplets, the moving object in the image cannot be tracked correctly. .

  Therefore, in Patent Document 1, as a method for determining whether or not water droplets are attached to the front surface of the imaging unit, changes in the shape and texture of an object estimated to be reflected in the input image and the input image are quantified. Is disclosed. In this method, the shape and texture of an object in a captured input image are compared with the shape and texture of an object in a background image acquired in advance, and the presence or absence of water droplets is determined based on these changes.

JP 2007-228448

  However, if the change in shape or texture with respect to the background image is used for judgment, the shape or texture changes even in the change area caused by the moving object to be tracked. Is difficult. In addition, since the change of the processing parameter of the obstacle detection program when the water droplet adhesion is detected is performed after a predetermined number of frames have passed, the water droplet is also subject to the tracking process until then. For this reason, when water droplets adhere to the imaging unit one after another, such as during rain, the processing load increases explosively. However, an apparatus that does not have abundant hardware resources, such as an interphone with a camera, and that can acquire and process images only for about 2 frames per second, for example, cannot increase the processing amount. In addition, the possibility of misjudging water droplets as a tracking target increases, which may lead to deterioration in tracking performance.

  Therefore, the present invention provides a moving object tracking device capable of reducing the processing load caused by water droplets by accurately excluding the change area due to the adhesion of water droplets from the processing target in the tracking processing of the moving object in the image. The purpose is to provide.

  In order to solve the above problems, a moving object tracking device according to the present invention includes an image input unit that sequentially acquires images, moving object information used for tracking for each moving object, a reference image that does not include a moving object, And a tracking processing unit for sequentially processing an input image and tracking a moving object on the image. In such a moving object tracking device, the tracking processing unit performs first processing from a change area extraction unit that extracts a change area by performing a difference process between the reference image and the input image, and an enlarged change area obtained by expanding the change area of the reference image. Extracting feature information and extracting second feature information from the change area of the input image, or extracting first feature information from the change area of the reference image and contracting the change area of the input image A feature extraction means for extracting second feature information from the region, a similarity between the first feature information and the second feature information is obtained, and if the similarity is equal to or greater than a certain value, the change region is input to the image input unit A water droplet determination unit that determines a water droplet change region caused by a water droplet attached to the water droplet and determines that the change region is a non-water droplet change region caused by a moving object other than a water droplet when the similarity is less than a certain value; and an input image Non-water droplet change region Means for associating the non-water droplet change area with the moving object information using the feature information extracted from the feature information included in the moving object information and the correspondence between the non-water drop change area and the moving object information And moving object information updating means for updating moving object information associated with the non-water droplet change area based on the attachment.

  As described above, the moving object tracking device according to the present invention can reduce the processing cost by excluding the change area determined as the water drop change area caused by the water drop from the tracking target. In particular, this moving object tracking device extracts feature information from a region in which a change region in a reference image or an input image is expanded or contracted in consideration that a portion to which a water droplet has adhered is reduced and captured due to a lens effect. This makes it possible to compare feature information for image regions that are estimated to correspond to each other when water droplets adhere to the reference image and the input image, so it is possible to determine whether or not the region is a water droplet change region with higher accuracy. can do.

  In the moving object tracking device according to the present invention, the feature extraction means obtains a plurality of enlarged change areas expanded at different ratios, extracts respective first feature information from the plurality of enlarged change areas, and a water droplet determination means. It is preferable to determine the similarity between each of the first feature information and the second feature information, and determine whether the region is a water droplet change region based on the highest similarity among the respective similarities. .

  In the moving object tracking device according to the present invention, the feature extraction means extracts the first feature information by multiplying a reference image by a weighting factor that decreases in value according to the distance from the center of the change area. Is preferred.

  Further, in the moving object tracking device according to the present invention, the feature extraction means obtains a plurality of reduced change areas contracted at different ratios, and extracts each second feature good news from the reduced change areas, The water droplet determining means obtains the similarity between the first feature information and the second feature information, and determines whether or not the region is a water droplet change region based on the highest similarity among the similarities. It is preferable to do.

  In the moving object tracking device according to the present invention, the feature extraction means extracts the second feature information by multiplying the input image by a weighting factor that increases in value according to the distance from the center of the change area. Is preferred.

  Furthermore, in the moving object tracking device according to the present invention, it is preferable that the first feature information and the second feature information are luminance histograms or color histograms.

  The moving object tracking device according to the present invention can reduce the processing load caused by water droplets by accurately excluding the change area due to the attachment of water droplets from the processing target in the tracking processing of the moving object in the image.

  As a preferred embodiment of the present invention, the moving object tracking device according to the present invention is applied to an interphone with a camera, and an image of a suspicious person who stays for a long time within an angle of view of a camera installed outdoors in a substantially horizontal direction. A description will be given based on an example of detection by processing.

  The moving object tracking device according to the present embodiment excludes, from the tracking target, a change area that is determined to be a water drop change area that is generated when a water drop adheres to the imaging device. In particular, the moving object tracking device extracts feature information from an enlarged change area obtained by expanding a change area of the reference image and a change area of the input image. When the feature information extracted in this way is similar to each other, it is determined that the change area is a water drop change area.

  FIG. 1A is a diagram showing an installation example of an interphone with a camera on which a moving object tracking device according to an embodiment of the present invention is mounted. As shown in the example of FIG. 1A, the camera intercom 1 is installed on a wall surface 4 such as a side of the entrance door 5. As shown in FIG. 1B, the camera 2 built in the intercom with camera 1 is installed with the optical axis oriented in a substantially horizontal direction, and continuously images the monitoring area in front of the entrance. The moving object tracking device connected to the camera 2 tracks a moving object in the image while sequentially receiving images continuously captured by the camera 2, so that the suspicious person stays within the angle of view of the camera 2 for a long time. Etc. are detected.

  FIG. 2 is a block diagram illustrating a configuration of the camera intercom 1 on which the moving object tracking device 10 according to the embodiment of the present invention is mounted. As shown in FIG. 2, the moving object tracking device 10 according to an embodiment of the present invention includes an image acquisition unit 110, a tracking processing unit 130, a storage unit 140, a stay determination unit 150, a person determination unit 160, The alarm unit 170 is connected to the imaging device 20.

  The imaging device 20 such as the camera 2 mounted on the camera-equipped interphone captures a predetermined monitoring area at predetermined time intervals, and sequentially sends the captured images to the tracking processing unit 130. Hereinafter, the unit of time recorded in the predetermined time is referred to as time. As the imaging device 20, for example, a known device such as an internal camera or an external camera configured to include an imaging device such as a CCD device or a C-MOS device, an optical system component, an A / D converter, or the like is used. it can. Moreover, the resolution of the imaging device 20 can be selected according to a specific application such as a tracking target. In addition, as a wavelength band used for image capturing, it is preferable to select a visible light wavelength or an infrared wavelength according to a specific application such as a tracking target.

  The image acquisition unit 110 is an interface for receiving image signals sequentially transmitted from the imaging device 20, and receives images from the imaging device 20 via wired or wireless wiring such as SCSI, USB, LAN, and dedicated cables. The image acquisition unit 110 sends the image received from the imaging device 20 to each unit of the moving object tracking device 10.

  The storage unit 140 can store various programs and various data. For example, a semiconductor memory such as RAM or ROM, EPROM, a magnetic recording medium such as a hard disk, an optical recording medium such as a CD-ROM, DVD-R / W, or the like. Can be used. The storage unit 140 is connected to the image acquisition unit 110, the tracking processing unit 130, the stay determination unit 150, the person determination unit 160, and the alarm unit 170, and reads and writes various programs and various data according to requests from each unit. . The storage unit 140 stores moving object information 141, a reference image 142, water droplet information 143, and a suspicious person detection image 144.

  The moving object information 141 is information relating to past moving objects acquired for each unit time stored for each moving object being tracked.

  The moving object information 141 includes, for example, feature information of the moving object, a template, a region of the moving object in the input image one time ago, a movement amount of a predetermined number in the past (for example, five times), and the moving object on the screen. The time that has elapsed since appearing (hereinafter referred to as existing time) is included.

  The feature information is information representing the feature of the moving object. In the present embodiment, for example, the luminance histogram and area of the image of the moving object are included, but the average or standard deviation of pixel values, edge information, texture information, and the like are included. Other feature information such as a color histogram may be included. Here, a description will be mainly given of an embodiment in which a luminance histogram having a relatively small processing load is used as the permeability characteristic information that hardly changes even through water droplets. However, other feature information such as a color histogram and a fractal dimension can be used as long as the feature information does not change much even when photographed through a water droplet. The luminance histogram will be described in detail in the description of the processing of the feature extraction unit 1313 described later.

  The template used in the present embodiment is a template image of a moving object detected in the input image one time before, and a template for specifying where the moving object being tracked is located in the input image at the current time. Used in matching. Therefore, any type of template such as various feature vectors and eigenvectors may be used as long as it can be used as a template for template matching.

  The area of the moving object is a range that the moving object occupies in the input image, and can generally be represented by its shape, size, and position. In this embodiment, a circumscribed rectangle is used as the area, and the area can be specified by the size and position of the circumscribed rectangle. For example, the position of the center of gravity may be used.

  The reference image 142 is an image that serves as a reference for comparison when extracting the change area. For example, a background image when a moving object does not exist in the monitoring area is captured in advance and stored in the storage unit 140. .

  As will be described later, the water drop information 143 is obtained when the extracted change area is labeled with the center of gravity position, shape, and labeling of the water drop change area determined by the water drop determination unit 1314 as a result of the water drop. The label is stored as necessary.

  The suspicious person detection image 144 stores an image of a moving object that seems to be a suspicious person when a moving object that has been determined to be a person who has been in the screen for more than a certain time as a result of tracking the moving object in the image is detected. Is.

  The tracking processing unit 130 processes the images sequentially received from the image acquisition unit 110 to track a moving object in the image. Similar to the stay determination unit 150, the person determination unit 160, and the alarm unit 170, the tracking processing unit 130 reads a program describing the processing procedure stored in the storage unit 140, and uses a processing device such as a CPU, DSP, or MCU. It is realized by executing. The tracking processing unit 130 is connected to the image acquisition unit 110, the storage unit 140, the stay determination unit 150, the person determination unit 160, and the alarm unit 170.

  As shown in FIG. 2, the tracking processing unit 130 includes a change area extraction unit 131, a feature extraction unit 1313, a water drop determination unit 1314, a water drop region removal unit 1315, an association unit 132, and a moving object information extraction unit. 133 and moving object information updating means 134.

  The change area extraction unit 131 includes a reference image 142 that is stored in advance in the storage unit 140 and has no moving object, and an image that is captured by the imaging device 20 and received via the image acquisition unit 110 (hereinafter referred to as an input image). And a change area in the input image is extracted. And the change area extraction means 131 labels each change area extracted by labeling.

  The feature extraction unit 1313 calculates feature information by calculating a reference image in the change region, a luminance histogram of the input image, and the like for each change region extracted by the change region extraction unit 131. At this time, the feature extraction unit 1313 extracts feature information from a region in which the change region is expanded in a predetermined ratio in the reference image. Alternatively, the feature extraction unit 1313 may extract feature information from an area in which the change area is contracted in a predetermined ratio or the like in the input image.

  The water drop determination means 1314 compares the brightness histogram in the change area of the reference image and the input image based on the brightness histogram extracted by the feature extraction means 1313 as the transmission characteristic information that hardly changes due to the attachment of the water drop, and the change area extraction means 131. It is determined whether each change area extracted by is a water drop change area due to a water drop.

  The water droplet area removing unit 1315 removes, from the change region to be processed, the water droplet change region determined by the water droplet determining unit 1314 from the change region extracted by the change region extracting unit 131. Further, the water droplet region removing unit 1315 may store information regarding the water droplet change region as the water droplet information 143.

  The associating unit 132 selects each change region to be processed based on the degree of correspondence between the feature information of each changed region extracted by the feature extracting unit 1313 and the moving object information 141 of each moving object stored in the storage unit 140. The most likely moving object that exists in the change area is associated.

  The moving object information extraction unit 133 performs template matching and the like based on the change area associated with the association unit 132 and the moving object information 141, and moves object information regarding each moving object, that is, the moving object in the input image. Extract position, area, feature information, etc.

  The moving object information update unit 134 updates the moving object information 141 stored in the storage unit 140 using the moving object information extracted by the moving object information extraction unit 133. Also, the existence time of the moving object information 141 is updated for the moving object existing at the current time.

  The stay determination unit 150 checks the existence time stored in the storage unit 140 for each moving object. If the stay determination unit 150 exists in the image longer than a predetermined time (for example, one minute), the stay determination unit 150 determines whether it is a human. Information on the area of the moving object is sent to the person determination unit 160 for examination.

  The person determination unit 160 determines whether or not the moving object that has been present in the screen for a longer time than the predetermined time sent from the stay determination unit 150 is a human, and the determination that the person is a human is an alarm. The result is sent to the unit 170. Further, the image at that time may be stored in the storage unit 140 as the suspicious person detection image 144.

  When the alarm unit 170 receives the result from the person determination unit 160, the alarm unit 170 notifies the user that a suspicious person has been detected by an alarm lamp such as an LED or an alarm buzzer. A message stored in advance may be played as a voice through a speaker or displayed on a display. Moreover, the suspicious person detection image 144 memorize | stored in the memory | storage part 140 can also be output via output devices (not shown), such as a display and a printer. Further, the alarm unit 170 may warn the user that the number of water drops has reached a certain level based on the water drop information 143.

[Operation of moving object tracking device]
FIG. 7 is a flowchart showing an operation flow of the moving object tracking device 10 according to the embodiment of the present invention. The details of the operation of each part of the moving object tracking device 10 will be described with reference to FIG.

  In step S <b> 100, the moving object tracking device 10 stores, in the storage unit 140, a background image captured in advance in a situation where the moving region does not exist within the angle of view of the imaging device 20 by the imaging device 20 as the reference image 142. , Initialization for moving object tracking processing such as setting the moving object information 141 regarding the moving object to be tracked to an initial state is performed.

  In step S <b> 200, the moving object tracking device 10 receives images captured by the imaging device 20 at predetermined time intervals via the image acquisition unit 110 and passes them to the tracking processing unit 130. In this way, the moving object tracking device 10 repeats the processing of steps S200 to S1100 by the tracking processing unit 130 every time input images are sequentially sent from the imaging device 20 at regular time intervals.

  In step S <b> 300, the tracking processing unit 130 compares the input image newly received from the imaging device 20 with the reference image 142 stored in the storage unit 140 by the change region extraction unit 131, and changes a region having a difference. Extract as a region.

  For example, the change area extraction unit 131 performs difference processing on pixel values of corresponding pixels in the input image and the reference image 142, performs binarization on a pixel group having a difference value equal to or greater than a certain value, and labels the extracted areas. I do. In this way, a change area different from the reference image 142 in the input image, that is, an area where a moving object is estimated to exist is extracted. Note that various known methods can be used for the difference processing, binarization, and labeling methods, and thus detailed description thereof is omitted.

  In the change area determination process in step S350, the tracking processing unit 130 removes the water drop change area determined as a water drop by the water drop determination means 1314 from the change area extracted by the change area extraction means 131 in step S300. It is removed from the processing target by 1315. Note that the change area determination processing in step S350 will be described in detail later.

  In step S <b> 400, the tracking processing unit 130 uses the association unit 132 to move the moving object information 141 of the moving object stored in the storage unit 140 and the non-water droplet change left as a processing target in the change area determination process in step S <b> 350. Associate with an area.

  The associating means 132 includes moving object information 141 relating to the tracking-target moving object detected from the input image one hour before stored in the storage unit 140, and the feature extracting means 1313 uses each change area of the input image at the current time. Based on the feature information extracted from the image and the similarity calculated on the basis of the feature information, the moving object detected one hour ago is associated with the change area of the input image at the current time that is most likely to exist. .

  FIG. 3 is a diagram illustrating an example of one-to-one association between a moving object one hour before and a change area at the current time according to an embodiment of the present invention. With reference to FIG. 3, the process of the matching means 132 is demonstrated concretely. FIG. 3A is a diagram showing a moving object extracted from the input image 300 acquired at the previous time. Each moving object at the previous time is labeled with ID1 and ID2, respectively. FIG. 3B is a diagram illustrating a change area extracted from the input image 310 acquired at the current time. Each change area at the current time is labeled with IDA and IDB, respectively.

  FIG. 3C is a diagram showing a correspondence table 320 indicating the degree of correspondence between each moving object at the previous time and each change area at the current time. In the correspondence table 320, the labels ID1 and ID2 of each moving object at the previous time correspond to each row, the labels IDA and IDB of each change area at the current time correspond to each column, and each moving object and the change area Correspondence levels are arranged in a matrix. As the degree of correspondence, the similarity between the feature amount obtained from the moving object information corresponding to each moving object at the previous time and the feature amount obtained from the feature information of each change area at the current time extracted by the feature extraction unit 1313 Use degrees. In the present embodiment, the similarity is normalized so that it is 1.0 when they are exactly the same, a positive number of 1.0 or less when there is some common point, and 0 when there is no common point. Yes. Furthermore, in the correspondence table 320, one row indicating the new appearance degree obtained by subtracting the maximum value of each column from 1.0 and one column indicating the lost degree obtained by subtracting the maximum value of each row from 1.0 are added. ing. In addition, regarding the correspondence table 320, since various known correspondence tables can be used, detailed description thereof is omitted.

  In the correspondence table created in this way, by searching for the combination of the moving object and the change area that maximizes the total degree of correspondence, the correspondence between the label of the moving object at the previous time and the label of the change area at the current time To do. For example, an optimization method such as the Hungarian method can be used to obtain a combination of correspondences that maximizes the sum of correspondences based on the correspondence table. Details of the Hungarian law are described in, for example, Hiroshi Sadamichi, “Management Science”, Ohmsha, 1989.

  In the example of FIG. 3, the previous time label ID1 is associated with the current time label IDA, and the previous time label ID2 is associated with the current time label IDB. The moving object is tracked by sequentially repeating this process at each time.

  As shown in FIG. 3C, the correspondence table has a size of (number of labels at the previous time + 1) rows x (number of labels at the current time + 1) columns. In particular, the process of associating the label with the previous time and the current time is performed for each combination of labels, so that the processing cost increases exponentially as the number of labels increases. For this reason, it is important to reduce the number of labels to be processed by deleting unnecessary labels in advance. In this embodiment, the label of the change area determined to have been caused by water droplets is deleted from the label to be processed as an unnecessary label.

  FIG. 4 is a diagram illustrating an example of one-to-one correspondence between a moving object including a water droplet one hour before and a change region including a water droplet region at the current time according to an embodiment of the present invention. For example, in the input image 400 at the previous time shown in FIG. 4A, water drops 402 to 404 are shown in addition to the human 401, and labels ID1 to ID4 are respectively attached as shown in FIG. 4B. The four moving objects are extracted. Then, as shown in FIG. 4C, if the water droplet 415 is further increased by one in the input image 410 at the current time, five change areas are extracted and five label IDAs are extracted as shown in FIG. ~ IDE is attached.

  In this case, when the correspondence table is created, the correspondence table 420 of 5 rows and 6 columns is formed as shown in FIG. 4E, and the processing load is significantly larger than the correspondence table 320 of 3 rows and 3 columns of FIG. To increase. However, when the label determined to be due to the water droplet is removed by the water droplet region removing means 1315, only one of ID1 and IDA remains at the previous time and the current time. In this case, the created correspondence table is small in 2 rows and 2 columns (only in the thick frame 421 of the correspondence table 420 in FIG. 4E), and the processing cost of the association removes the label of the water drop change area. Compared to when not, it can be dramatically reduced.

  In addition, by minimizing the number of labels to be processed, it is possible to reduce the possibility of incorrect association, so there is also an advantage that contributes to an improvement in tracking performance.

  In step S410, the tracking processing unit 130 moves the moving object of each moving object in the input image at the current time based on the moving object information 141 by the moving object information extracting unit 133 based on the change area and the moving object information 141 associated with each other. Extract information.

  For example, the moving object information extraction unit 133 extracts information such as the area of the moving object and the amount of movement based on the area of each moving object specified by performing template matching or the like. Further, for example, the moving object information extraction unit 133 extracts feature information used for tracking processing, such as a luminance histogram and an area of the moving object image, from the input image of the moving object region.

  In step S <b> 420, the tracking processing unit 130 updates the moving object information 141 using the moving object information extracted by the moving object information extracting unit 133 by the moving object information updating unit 134.

  In step S600, the moving object information updating unit 134 increases the number of tracking successes of the moving object information 141 stored in the storage unit 140 by one for the moving object that has been successfully tracked. Alternatively, the existence time may be increased by the shooting interval. On the other hand, if tracking fails and it is lost, the number of tracking successes or the existence time is reset to zero. When it is determined that the moving object has moved out of the screen, the moving object information update unit 134 deletes the moving object information 141 regarding the moving object from the storage unit 140.

  In step S700, the moving object tracking device 10 determines whether or not the moving object successfully tracked by the tracking processing unit 130 exists in the image for longer than a predetermined time by the stay determination unit 150.

  For example, the stay determination unit 150 determines whether or not the moving object has been present in the image longer than a predetermined time by determining whether or not the tracking success number of the moving object is greater than a predetermined number. Determine whether. In addition, the stay determination unit 150 updates the existence time for the moving object present in the input image acquired at the current time, stores the existence time in the storage unit 140, and the moving object exists for a long time. It may be used to determine whether or not it exists.

  When the moving object exists in the image longer than the predetermined time (Yes in Step S700), in Step S800, the moving object tracking device 10 determines whether or not the moving object is a person by the person determination unit 160. Determine.

  The person determination unit 160 determines whether or not the moving object that has been determined to be present in the screen for a longer time than the predetermined time by the stay determination unit 150 is a human, and determines that the moving object is a human (in step S800). Yes) outputs the result to the alarm unit 170 (S900). Then, the moving object image is stored in the storage unit 140 as the suspicious person detection image 144. Further, for example, other information such as a time period during which the suspicious person stays may be stored in the storage unit 140 together. Further, for example, when a user requests display of such information via an input device such as a switch button or a touch panel in a base unit of an interphone installed indoors, the storage unit 140 is accessed via an output device such as a display. The stored information may be displayed.

  Various known methods can be used as a method for determining whether or not the moving object is a human in the human determination unit 160. For example, P. Viola, M. Jones & D. Snow, “Detecting Pedestrians Using Patterns of Motion and Appearance”, IEEE International Conference on Computer Vision, pp.734, 0ct, As described in 2003, it is possible to determine whether or not a person is a human by a cascade type discriminator based on Haar-Like features learned by AdaBoost using human images of various appearances taken in advance.

  When it is determined that the moving object is a human (Yes in Step S800), in Step S900, the moving object tracking device 10 warns the user with light or sound by the alarm unit 170, and is stored in the storage unit 140. The detected suspicious person detection image 144 is displayed on an external device, for example, a monitor of an interphone master unit installed indoors.

  In step S1100, for example, when the number of attached water droplets or the ratio of the area occupied by water droplets is equal to or greater than a threshold based on the water droplet information 143 stored in the storage unit 140, the tracking accuracy is likely to decrease. The user may be warned via the alarm unit 170, and the tracking process may be stopped if necessary.

  FIG. 8 is a flowchart showing the processing flow of one embodiment of the change area determination in step S350 of FIG. Details of the first embodiment of the change area determination process will be described with reference to FIG.

  The tracking processing unit 130 repeatedly executes the processes of steps S303 to S309 for each change area label extracted by the change area extraction unit 131 in step S300.

  In step S304, the feature extraction unit 1313 performs image processing on the input image in the change area to which the label is attached, and calculates a feature amount. Examples of the feature amount include a circumscribed rectangle of the change area, the number of pixels, a centroid position, and a luminance histogram. Here, as an example, a luminance histogram in the change region of each label is calculated. The luminance histogram is information that is aggregated over the entire change region, and is one piece of transmissive characteristic information that does not change much even if water droplets adhere to the imaging device 20. Accordingly, if the similarity between the luminance histogram in the change area in the input image and the luminance histogram in the change area in the reference image is high, it can be determined that the change area is due to the attachment of water droplets.

  The luminance histogram is a feature amount representing the luminance distribution in the object image. After classifying which region of the luminance each pixel in the change region belongs to by a predetermined width and accumulating the frequency, Calculated by dividing by the total number of pixels and normalizing. For example, in an 8-bit, 256-gradation monochrome image, the luminance is divided into eight regions at equal intervals, the number of pixels belonging to each region is accumulated, and then divided by the total number of pixels. As described above, the luminance histogram is feature information having a low processing cost compared to the shape, texture, and the like.

  In step S305, the feature extraction unit 1313 expands the change area in the reference image of the target label to the enlargement change area estimated to be enlarged by the water droplet. As shown in FIG. 5, the change area is expanded in order to cope with a phenomenon in which when a water droplet adheres to the front surface of the imaging device 20, a wider range of background is reflected by the lens effect on the attached portion. An explanation will be given below with an example. Incidentally, regarding the expansion process, various known image processing methods such as expansion operation of morphological operation can be used, and thus detailed description thereof is omitted.

  FIG. 5 is a schematic diagram illustrating the lens effect of water droplets attached to the front surface of the imaging device 20. FIG. 5A is a schematic view of the imaging device 20 when no water droplets are attached, and FIG. 5B is a top view of the imaging device 20 when water droplets are attached. It is a schematic diagram. Here, the point F represents the focal point of the imaging device 20, the line segment P represents the camera cover, and the solid line extending radially from the focal point F represents the angle of view. As shown in FIG. 5 (a), the angle of view when no water droplet is attached is the angle θ. However, as shown in FIG. 5 (b), when the water droplet adheres, the water droplet acts like a convex lens. The angle of view spreads due to refraction and becomes θ ′. Due to this lens effect, the background of the water droplet portion is reduced in the input image.

  FIG. 5C is a schematic diagram of the reference image 500 obtained when no water droplet is attached, and shows a case where a tree 501 is reflected. FIG. 5D is a schematic diagram of the input image 510 obtained when a water droplet is attached, and the change area extraction unit 131 extracts the outline 512 of the water drop as a change area. As shown in FIG. 5D, the tree 501 in the reference image 500 shown in FIG. 5C reflects the entire tree 513 in a small amount inside the change area 512 due to water droplets in the input image 510. .

  In FIG. 5 (e), an area that matches the change area 512 in the input image 510 in FIG. 5 (d) in the reference image 500 in FIG. 5 (c) is shown as a change area 502. An image of the whole tree 501 shown in the enlarged change area 504 indicated by a broken line in FIG. 5 (e) is an image of the tree reduced and shown in the change area 512 of the input image 510 in FIG. 5 (d). It almost matches. Accordingly, if the luminance histogram is obtained from the change area 512 in the input image 510 and the luminance histogram is obtained from the enlarged change area 504 corresponding to the change area 512 of the input image 510 in the reference image 500, the reference image 500 and It is possible to compare luminance histograms of image regions that are approximately equal to the input image 510.

  Therefore, in step S306, the feature extraction unit 1313 calculates a luminance histogram of the reference image 500 in the enlarged change area 504 obtained by expanding the change area 502, as shown in FIG. In step S307, the similarity between the brightness histogram calculated in this way and the brightness histogram of the input image 510 in the change area 512 calculated in step S304 is calculated. If the similarity is equal to or greater than a predetermined threshold, the change area of the label is determined. It is determined that the region is a water droplet change region.

  Regarding the ratio of expanding the label area, an appropriate value may be determined in advance, or the maximum ratio of expansion for the area is determined and gradually increased to the maximum ratio (for example, 10%) without changing the shape (for example, Steps S304 to S306 may be repeated so that the similarity of the luminance histogram is calculated for each size, and the largest value among the obtained values is adopted as the final similarity. .

  In addition, the image reflected in the water droplet usually becomes an image that is reduced as the distance from the center of gravity of the water droplet is reduced, such as a tree 513 that is reduced and projected by the water droplet in FIG. Therefore, if the luminance histogram is calculated as it is, the similarity with the luminance histogram calculated from the reference image may be lowered. Therefore, when calculating the luminance histogram, the appearance frequency in each luminance range of the luminance histogram may be accumulated while weighting according to the distance from the barycentric position of the label area.

  FIG. 5G is an example of a graph showing weighting according to the center of gravity C and the distance from the center of gravity C. For example, when the luminance histogram is calculated with equal weighting for the change region 512 of the input image 510 shown in FIG. 5D in step S304, when calculating the luminance histogram of the reference image to be compared in step S306, In the enlarged change area 504 shown in FIG. 5E, the frequency is accumulated while weighting so as to monotonously decrease as the distance from the center of gravity C increases as shown in FIG. 5G. This makes it possible to compare luminance histograms with higher accuracy because the closer to the periphery, the smaller the effect of the luminance value on the calculation of the luminance histogram. become.

  Note that when calculating the luminance histogram of the input image 510 in step S304, the appearance frequency is accumulated while weighting as shown in FIG. 5 (h) according to the distance from the center of gravity of the change area 512. May be.

In step S307, the water drop determination unit 1314 determines whether or not the change area is caused by the adhesion of water drops by comparing the luminance histograms in the change area of the input image and the reference image. The similarity between the two luminance histograms is expressed by Equation (1), where h (i) is the luminance histogram of the change area of the input image, g (i) is the luminance histogram of the change area of the reference image, and i is the bin number. Can be calculated.

  Since the luminance histograms h (i) and g (i) are normalized so that the total sum of bins is 1, the similarity according to Equation 1 is 1.0 when the luminance histograms completely match, In some cases, the positive number is less than 1.0, and the larger the value is, the larger the value is. The water drop determination unit 1314 determines that a change area having a value equal to or greater than a certain value in the luminance histogram similarity is a water drop change area.

  In step S308, when the water drop determination unit 1314 determines that the water drop change area is a water drop change area (Yes in step S307), the water drop area removal unit 1315 excludes the label of the change area from the target of the tracking process. Information on the change area may be stored in the water drop information 143 of the storage unit 140.

  FIG. 6 is a diagram illustrating an example of removing the water droplet change area from the extracted change area. A human 601 and water droplets 602 to 604 are shown in the input image of FIG. As shown in FIG. 6B, the change area extraction unit 131 extracts the change area by comparing the input image of FIG. 6A with the reference image, and identifies the person 601 of FIG. 6A as the label ID1. The water droplets 602, 603, and 604 in FIG. 6A are referred to as labels ID2, ID3, and ID4.

  Next, it is determined whether or not each label is a water droplet by the water droplet determination means 1314, and the labels determined as water droplets (ID2, ID3, ID4 in FIG. 6C) are excluded from the subsequent processing targets, and the water droplet information It is stored in the storage unit 140 as 143. The remaining change area (ID1 in FIG. 3D) becomes a subsequent processing target as a moving object to be tracked.

  FIG. 9 is a flowchart showing a processing flow of another embodiment of the change area determination in step S350 of FIG. With reference to FIG. 9, the second embodiment of the change area determination process will be described with respect to differences from the first embodiment shown in FIG. In the second embodiment, steps S310 to S312 shown in FIG. 9 are executed instead of steps S304 to S306 of the first embodiment shown in FIG.

  In step S310, the feature extraction unit 1313 calculates the feature amount of the region having the same size without being expanded with respect to the changed region extracted in step S301 in the reference image. That is, in FIG. 5E, a luminance histogram is created from the change area 502 of the reference image 500.

  In step S311, the feature extraction unit 1313 contracts the change area in the input image of the target label to a reduction change area estimated to be reduced by the water droplet. That is, a reduced change area 515 indicated by a broken line in the input image 510 in FIG.

  This is because the background image shown in the change area 502 of the reference image 500 shown in FIG. 5E is reduced and shown in the reduction change area 515 of the input image 510 shown in FIG. Because. Therefore, by contracting the change area for obtaining the luminance histogram from the input image 510 into the reduction change area 515 in FIG. 5F, the input image and the reference image can be compared with the luminance histogram of the substantially equal image area. .

  Therefore, in step S312, the feature extraction unit 1313 calculates a luminance histogram of the input image 510 in the reduced change area 515 in which the change area 512 due to the water droplet is contracted, as shown in FIG. In step S307, the similarity between the brightness histogram thus calculated and the brightness histogram of the reference image 500 in the change area 502 calculated in step S310 is calculated. If the similarity is equal to or greater than a predetermined threshold, the change area of the label is determined. It is determined that the region is a water droplet change region.

  An appropriate value may be determined in advance for the rate of shrinking the label area, or the maximum rate of shrinking for the area may be determined and gradually increased to the maximum rate (eg 10%) without changing the shape (eg 10%). Steps S310 to S312 may be repeatedly performed so that the similarity of the luminance histogram is calculated for each size, and the largest value among the obtained values is adopted as the final similarity. . As for the contraction process, as in the expansion process, various known image processing methods such as a contraction calculation of a morphological operation can be used, and thus detailed description thereof is omitted.

Similarly to step S304 in FIG. 8 in the case of the first embodiment, when the feature extraction unit 1313 extracts feature information from the input image in step S312 in FIG. 9, the weighting shown in FIG. May be performed. Alternatively, as in step S306 in FIG. 8, when the feature extraction unit 1313 extracts feature information from the reference image 142 in step S310 in FIG. 9, the weighting shown in FIG.
The other processing is the same as that in the case of FIG.

  In the above description, it is based on luminance information. However, if a color-compatible device such as a camera is used, it is possible to determine a water droplet using color information. The color histogram of the background image is information that hardly changes even through water droplets, and is clearly different if a moving object exists. Therefore, even if the load of color information processing is slightly increased compared to the luminance information processing, the accuracy of water droplet determination can be further improved.

In this case, the color histogram similarity shown in Expression (2) is used instead of the luminance histogram similarity in Expression (1). Here, h (y, u, v) is a color histogram of the label area of the input image, and g (y, u, v) is a color histogram of the label area of the reference image.

  As described above, according to the present invention, the change area generated in the image does not depend on the processing load such as the shape and texture, and the luminance information of the background image and the input image in the change area is compared. In addition, it is possible to determine water droplets in an image with a relatively small processing load.

  Considering that the water droplet portion is distorted and reduced in the input image due to the lens effect, the change area is expanded or contracted in either the input image or the reference image, thereby aligning image information to be compared. Therefore, even if feature information with a low processing cost such as a luminance histogram is used, the accuracy of water droplet determination can be sufficiently increased.

  Furthermore, since the number of processing objects can be significantly reduced by removing the change area due to the attachment of water droplets from the change area in the input image, the processing load such as association performed by combining these processing objects is reduced. It can be greatly reduced. In addition, the possibility of erroneously determining a water droplet as a moving object such as a person is reduced, and the tracking accuracy of the moving object can be further improved.

  Therefore, according to the present invention, in an environment where hardware resources are limited such as an interphone with a camera, an imaging device (camera) is installed outdoors, and it is unavoidable that water droplets adhere to the front surface due to rain or the like. Even in the case of shooting an environment where frequent occurrences occur, it is possible to continuously and accurately track the target moving object without increasing the processing amount.

  In this embodiment, in order to explain the advantage that each moving object can be accurately tracked even in an environment where hardware resources such as an interphone with a camera of the present invention are limited, an example of an interphone with a camera has been described. The present invention is not limited to this embodiment. In particular, it is effective in an environment having other limited hardware resources as long as water droplets such as rain and steam adhere to the imaging apparatus. Further, the present invention may provide the same effect even in a high-performance image processing environment having abundant hardware resources depending on the characteristics of the moving object to be tracked, the image capturing conditions, and other situations. Therefore, the present invention can be implemented in various forms as long as the spirit is not changed.

  When the present invention is applied to an interphone with a camera, a suspicious person (stalker) who stays in front of the interphone for a long time can be accurately detected even in situations where water drops frequently adhere to the front surface of the imaging unit due to rain, etc. It can be detected.

It is a figure which shows the example of installation of the intercom with a camera by which the moving object tracking device by one Embodiment of this invention was mounted. It is a block diagram which shows the structure of the intercom with a camera by which the moving object tracking device by one Embodiment of this invention was mounted. It is a figure which shows the example of 1-to-1 correspondence with the moving object of one time ago, and the change area | region of the present time by one Embodiment of this invention. It is a figure which shows the example of 1 to 1 matching with the change area | region containing the water drop area | region of the present time and the moving object containing the water drop of 1 time before by one Embodiment of this invention. It is a schematic diagram explaining the lens effect of the water droplet adhering to the front surface of an imaging device. It is a figure which shows the example which removes a water droplet change area | region from the extracted change area | region by one Embodiment of this invention. It is a flowchart which shows the operation | movement flow of the moving object tracking apparatus 10 by one Embodiment of this invention. It is a flowchart which shows the processing flow of one Example of the change area | region determination in FIG.7 S350. It is a flowchart which shows the process flow of the other Example of the change area | region determination in FIG.7 S350.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Interphone with a camera 10 Moving object tracking apparatus 20 Imaging apparatus 110 Image acquisition part 130 Tracking process part 131 Change area extraction means 132 Association means 133 Moving object information extraction means 134 Moving object information update means 140 Storage part 141 Moving object information 142 Reference | standard Image 143 Water drop information 144 Suspicious person detection image 150 Stay determination part 160 Person determination part 170 Alarm part 1313 Feature extraction means 1314 Water drop determination means 1315 Water drop area removal means

Claims (6)

An image input unit for sequentially acquiring images;
A storage unit that stores moving object information used for tracking for each moving object and a reference image that does not include the moving object;
A tracking processing unit that sequentially processes the input image and tracks a moving object on the image;
A moving object tracking device comprising:
The tracking processing unit
A change area extraction means for extracting a change area by performing a difference process between the reference image and the input image;
First feature information is extracted from an enlarged change area obtained by expanding the change area of the reference image and second feature information is extracted from the change area of the input image, or the change of the reference image Feature extraction means for extracting first feature information from a region and extracting second feature information from a reduced change region obtained by contracting the change region of the input image;
A similarity between the first feature information and the second feature information is obtained, and when the similarity is equal to or greater than a predetermined value, the change region is a water drop change region generated by a water drop attached to the image input unit; Water drop determination means for determining, when the similarity is less than the predetermined value, the non-water droplet change area generated by a moving object other than the water drop, the change area;
Association means for associating the non-water drop change area with the moving object information using the feature information extracted from the non-water drop change area of the input image and the feature information included in the moving object information; ,
A moving object comprising: moving object information updating means for updating the moving object information associated with the non-water droplet change area based on the association between the non-water drop change area and the moving object information. Tracking device. The feature extraction means obtains a plurality of the enlarged change areas expanded at different ratios, extracts each of the first feature information from the plurality of enlarged change areas,
The water drop determination means obtains the similarity between each of the first feature information and the second feature information, and is the water drop change region based on the highest similarity among the respective similarities. The moving object tracking device according to claim 1, wherein it is determined whether or not.   The movement according to claim 1, wherein the feature extraction unit extracts the first feature information by multiplying the reference image by a weighting factor that decreases in value according to a distance from the center of the change area. Object tracking device. The feature extraction means obtains a plurality of the reduced change areas contracted at different ratios, extracts the second feature bulletin from each of the plurality of reduced change areas,
The water drop determination means obtains the similarity between the first feature information and the second feature information, and determines the water drop change region based on the highest similarity among the similarities. The moving object tracking device according to claim 1, wherein it is determined whether or not there is any.   5. The movement according to claim 1, wherein the feature extraction unit extracts the second feature information by multiplying the input image by a weighting factor that increases in value according to a distance from the center of the change area. Object tracking device.   The moving object tracking apparatus according to claim 1, wherein the first feature information and the second feature information are luminance histograms or color histograms.

Images