JP2018185724A - Device, program and method for tracking object using pixel change processing image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device capable of further reliably tracking a plurality of objects.SOLUTION: An object tracking device which uses a time series image group that can include a plurality of tracking objects and determines, based on an output from a discriminator where an image or an image area is input, a position of the tracking object in each image to track a tracking object, includes, as a feature of a device configuration: mask processing means that executes pixel change processing for changing to a pixel pattern where a feature of another tracking object is eliminated or reduced with respect to a processing object area, which is a processing object area in an image or an image area included in the image group, determined based on a position determined as a past point in time or as a correct position for another tracking object other than one tracking object; and image output means that outputs the image or the image region subjected to the pixel change processing to the discriminator for learning and/or discrimination of one tracking object. Also, the discriminator may include a plurality of convolutional neural networks for inputting images or image areas at a plurality of time points.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an object tracking technique for analyzing a time-series image group that can include a tracking target and tracking the target.

  Currently, for the purpose of monitoring, marketing and the like, a tracking technique is being put into practical use, which analyzes time-series image data captured and generated by a camera and sequentially determines the position of a moving tracking target in real space. As a tracking target, various objects such as a person and a vehicle can be set as long as the object can be photographed. In addition, a technique for sequentially tracking a plurality of objects in parallel has been developed and is being actively improved. With this technology, for example, it is intended to more accurately grasp the flow lines of a large number of shop assistants and customers who stay and move in the store.

  The target tracking technique using such a time series image group is generally based on (a) a discriminator that outputs information related to the position of the object while online learning of the appearance of the object to be tracked by using an image every moment. And (b) a classifier that does not perform apparent online learning.

  For example, Non-Patent Document 1 discloses (a) online learning type tracking technology. In this technique, whenever a frame at a new time is acquired, detection and tracking processing for each ID (identifier) assigned to the tracking target object are sequentially performed. Specifically, the appearance of an object related to each ID is learned online, and an apparent identification model is configured for each ID. Then, using the apparent identification model for each ID learned online in this way, the new detection result and the locus of the object related to the ID tracked so far are linked.

  On the other hand, Non-Patent Document 2 discloses a technique for tracking a single object, which is a type that does not perform online learning in (b). This technique uses deep learning, and an image region pair of an object image region at the previous time and a candidate image region cut out as a candidate from the periphery of the region corresponding to the region at the previous time in the current time image. Thus, tracking is performed by estimating an image area where an object exists at the current time.

S.-H. Bae and K.-J. Yoon., "Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning", Published in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference, 2014 , Pages 1218-1225 D. Held, S. Thrun, and S. Savarese, "Learning to track at 100 fps with deep regression networks", Cornell University Library, Subjects: Computer Vision and Pattern Recognition (cs.CV), Cite as: arXiv: 1604.01802, 2016

  However, in the conventional technology as described above, particularly when tracking a plurality of objects in parallel, a problem still cannot be solved.

  For example, since the technique described in Non-Patent Document 1 learns the appearance of an object online, when a plurality of objects exist in image data, a large amount of memory is consumed and the consumption is controlled. Have a problem with being difficult. For example, if this technology is applied in a situation where there are many people such as exhibition halls and public roads, and the people included in the video change every moment, the memory consumption becomes enormous, and finally the tracking process There is also a possibility that it will not be possible. Specifically, the person model to be learned continues to increase as the ID of the person to be tracked increases while performing analysis for tracking over a long period of time, resulting in a situation where memory resources are insufficient. It will end up.

  On the other hand, in the technique described in Non-Patent Document 2, it is not necessary to learn the appearance of an object online, so the problem of memory consumption as described above does not occur. However, Non-Patent Document 2 merely sets a single object as a tracking target, and does not assume tracking of a plurality of objects. Here, if the technique described in Non-Patent Document 2 is applied to tracking a plurality of objects as they are, the problem of memory consumption does not occur, but an apparently similar object exists near the object to be tracked. In this case, the possibility of a phenomenon called drift increases.

  Here, the drift is a phenomenon in which an image area related to an apparently similar object existing in the vicinity is regarded as a correct answer area and an incorrect object starts to be tracked. For example, this is a phenomenon that may occur frequently when a plurality of persons superimposed on each other in a video are targeted for tracking. However, Non-Patent Document 2 does not propose any method for dealing with this drift because a single object is a tracking target in the first place.

  Incidentally, when drifting a plurality of objects, this drift is not completely eliminated even in the object tracking using online learning, and a countermeasure is still desired.

  SUMMARY An advantage of some aspects of the invention is that it provides an apparatus, a program, and a method that can more reliably track a plurality of objects.

According to the present invention, a time-series image group that can include a plurality of tracking targets is used, and the position of the tracking target in each image is determined based on an output from the classifier that inputs the image or an image region in the image. A device for determining and tracking the tracking target,
A processing target region in an image included in the image group or an image region in the image and determined based on a position determined at a past time or as a correct answer for other tracking targets other than one tracking target Mask processing means for performing a pixel change process for changing to a pixel pattern in which the characteristics of the other tracking target are eliminated or reduced with respect to the target region;
There is provided an object tracking device having an image output means for outputting the image or the image region subjected to the pixel change processing to a discriminator for learning and / or identification of the one tracking target.

  As one embodiment of the pixel changing process in the target tracking device of the present invention, the mask processing unit is configured to apply at least one processing target area related to the other tracking target to the image or other than the tracking target in the image area. It is also preferable to change to a pixel pattern determined based on the image area or a predetermined pixel pattern.

  Further, as another embodiment of the pixel change process, the mask processing means preferably performs a process of blurring the appearance of the other tracking target with respect to the processing target area related to at least one other tracking target.

  Further, the mask processing unit may superimpose the processing target area related to the other tracking target on the tracking target area including the position determined for the one tracking target as still another embodiment of the pixel changing process. In the case of having an overlapping area, it is also preferable not to perform the pixel changing process on the overlapping area.

  In addition, the target tracking device according to the present invention may be configured such that when the processing target area related to the other tracking target has a superimposed area superimposed on the tracking target area including the position determined for the one tracking target, It is also preferable to further include a tracking target area determination unit that changes the tracking target area related to the tracking target to an area excluding the superimposition area.

  Furthermore, as one embodiment of the object tracking device according to the present invention, the image output means outputs the image or the image area subjected to the pixel change processing at at least one time point to the learned classifier, The target tracking device preferably further includes target position determining means for determining the position of the one tracking target at the one time point based on the output from the learned classifier.

  Further, in this embodiment, the image output means includes the image or the image area subjected to the pixel change process at one time point, and the pixel change process at at least one time point before the one time point. It is also preferable to output the image or the image region that has been subjected to learning to a learned classifier.

  Furthermore, the discriminator according to the present invention includes a plurality of first neural networks that input the image or the image region at each time point and output the feature information related to the feature of the image or the image region, and the plurality of the first neural networks. It is also preferable to include a second neural network that inputs a plurality of the feature information output from the neural network and outputs information related to the position of the one tracking target.

  The image output means preferably outputs the image corresponding to the feature information or information related to the time point of the image region in association with the feature information to the second neural network.

  Further, when the image output means learns the discriminator according to the present invention, at least the image or the image area subjected to the pixel change process, and the position of the tracking target area as a correct answer in the image or the image area It is also preferable to output a set of information related to the above to the discriminator.

  As still another embodiment of the pixel changing process in the object tracking device of the present invention, the mask processing means is configured to detect the discriminator according to the present invention, and the misaligned position from the determined processing target area. It is also preferable to perform pixel change processing on the processing target area.

  Furthermore, as another embodiment of the discriminator, the image output means outputs at least the image or the image area at one time point subjected to the pixel change processing to the discriminator that performs online learning, The target tracking device preferably further includes target position determining means for determining the position of the one tracking target at the one time point based on an output from the discriminator that is learning online.

Further, according to the present invention, a time-series image group that can include a plurality of tracking targets is used, and the tracking target of each image is determined based on an output from the classifier that inputs the image or an image area in the image. A program for operating a computer mounted on a device for determining a position and tracking the tracking target,
A processing target region in an image included in the image group or an image region in the image and determined based on a position determined at a past time or as a correct answer for other tracking targets other than one tracking target Mask processing means for performing a pixel change process for changing to a pixel pattern in which the characteristics of the other tracking target are eliminated or reduced with respect to the target region;
Provided is an object tracking program for causing a computer to function as an image output means for outputting the image or the image area subjected to the pixel change processing to a discriminator for learning and / or identification of the one tracking object. The

According to the present invention, a time-series image group that can include a plurality of tracking targets is used, and the tracking target of each image in the image is output based on the output from the identifier that inputs the image or the image region in the image. An object tracking method in a computer mounted on an apparatus for determining a position and tracking the tracking object,
A processing target region in an image included in the image group or an image region in the image and determined based on a position determined at a past time or as a correct answer for other tracking targets other than one tracking target Performing a pixel change process for changing to a pixel pattern in which the characteristics of the other tracking target are eliminated or reduced with respect to the target region;
And outputting the image or the image region subjected to the pixel change processing to a discriminator for learning and / or identification of the one tracking target.

  According to the object tracking device, program, and method of the present invention, a plurality of objects can be tracked more reliably.

1 is a schematic diagram illustrating an embodiment of an object tracking system including an object tracking apparatus according to the present invention. It is a functional block diagram which shows the function structure in one Embodiment of the object tracking apparatus by this invention. It is a mimetic diagram showing one embodiment of pixel change processing (mask processing) concerning the present invention. It is a schematic diagram which shows other embodiment of the pixel change process (mask process) which concerns on this invention. It is a schematic diagram which shows further another embodiment of the pixel change process (mask process) which concerns on this invention. It is a schematic diagram which shows one Embodiment of the discriminator which concerns on this invention. It is a schematic diagram which shows other embodiment of the discriminator which concerns on this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Object tracking system]
FIG. 1 is a schematic diagram showing an embodiment of an object tracking system including an object tracking apparatus according to the present invention.

The object tracking system of this embodiment shown in FIG.
(A) one or a plurality of cameras 2 capable of capturing an object to be tracked and capable of transmitting information of the captured image in time series via a communication network;
(B) A target tracking device 1 that uses a time-series image group acquired from the camera 2 via a communication network and predicts position information of the object to track the object.

  Here, various objects can be used as objects to be tracked, such as people, animals, vehicles, and other physical objects that can move. Also, the location where the image is taken is not particularly limited. For example, it may be outdoors where spectators, commuters, shoppers, workers, pedestrians, runners, etc. can be reflected, and it may be indoors such as inside a company, school, home, or store. it can.

  Incidentally, in the present embodiment, there are assumed an environment that has been a weak point of the prior art in which there are a plurality or many objects (persons) to be tracked, and they accompany, pass each other, or are superimposed in an image. .

  In addition, a communication network that is a transmission path for image information can be a wireless local area network (LAN) such as Wi-Fi (registered trademark), for example. Alternatively, the camera 2 and the target tracking device 1 are connected via the Internet via a wireless access network such as LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access) or 3G (3rd Generation). It may be.

  Further, the camera 2 and the target tracking device 1 may be connected for communication via the Internet via a fixed access network such as an optical fiber network or ADSL (Asymmetric Digital Subscriber Line), or via a private network. As a change mode, the camera 2 and the target tracking device 1 may be directly connected by wire. Furthermore, a camera control device (not shown) that can collect image information output from the plurality of cameras 2 and transmit the image information to the target tracking device 1 may be provided.

Similarly, as shown in FIG. 1, the target tracking device 1 is configured to output an image of the acquired time-series image group or an image area (image patch) in the image based on an output from the discriminator 11. A device for determining a position of a tracking target and tracking the tracking target,
(A) To an image included in a time-series image group or a “processing target region” in an image region in this image, to a pixel pattern in which features of other tracking targets other than one tracking target have been eliminated or reduced A mask processing unit 111 for performing a “pixel change process” for changing,
(B) An image output unit 113 that outputs an image or an image area subjected to the “pixel change process” to the classifier 11 for learning and / or identification of the one tracking target. Yes.

Here, the “processing target area” in (A) is determined based on a position determined at a past time or as a correct answer for other tracking targets other than this one tracking target. Furthermore, as a specific example of the “pixel change process” in (A) above,
(A1) The “processing target area” may be changed to an area having a predetermined pixel pattern. For example, it can be painted with a single color such as black. Also,
(A2) The “processing target area” may be changed to an area having a pixel pattern determined based on an image area other than the tracking target. For example, it is possible to determine a background color to be a background to be tracked and fill it with this background color. further,
(A3) A process for blurring the appearance of other tracking targets may be performed on the “processing target area”. For example, it is possible to blur with a color calculated from the surrounding colors.

  Naturally, the “pixel changing process” is not limited to the above-described forms (A1) to (A3), and changes to a pixel pattern in which other features to be tracked are eliminated or reduced. If there are, various processes can be employed.

  As described above, when tracking a plurality of objects as tracking targets, the target tracking device 1 performs the “pixel changing process” on other objects other than the one object that is the tracking processing target, thereby identifying the discriminator 11. The characteristics of other objects in the image or image area (image patch) input to the image are deleted or reduced. As a result, by using the image or the image region on which such processing has been performed, the classifier 11 is learned and / or the classifier 11 performs the classification process, thereby making this one object another object. It becomes possible to continue to specify more reliably without being confused with.

  In other words, by using the “image object mask method” of the present invention as described above, even if there is an apparently similar object near the object to be tracked, the drift that has been a problem in the past has been solved. It is possible to suppress the occurrence. Here, the drift is a phenomenon in which an image area related to a similar object existing in the vicinity is regarded as a correct area and an incorrect object starts to be tracked.

  By the way, the “image object mask method” of the present invention is suitable for tracking processing that does not perform online learning in order to solve the problem of increase in memory consumption (calculation cost). It can also be applied to the tracking processing to be performed. Among these, when applied to a tracking process for a plurality of objects that do not perform online learning, it is possible to achieve both suppression of increase in memory consumption (calculation cost) and suppression of occurrence of drift. On the other hand, when applied to a tracking process that performs online learning, drift can be further eliminated.

  In any case, the “image object mask method” of the present invention is a technique that suppresses mistracking and more reliably tracks a plurality of objects. In particular, under the non-online learning that is the former case described above. This is a very important technology for realizing multi-tracking.

  Note that the time-series image group handled by the apparatus 1 is not limited to image data generated by camera shooting as in the present embodiment. Various data are applicable as long as the data is related to the actual position and appearance of the tracking target. For example, depth value information (for each target pixel) generated by a depth camera can be used as image data.

  Further, in the present embodiment, the object tracking device 1 uses the world coordinate system Gx−, which is extended in the real space, to the position coordinates (u, v) in the image coordinate system uv that is stretched in the image acquired every moment. Information relating to the position in the real space is calculated from the image information of the tracking target object using a coordinate conversion operation for converting the position coordinates (gx, gy, gz) in Gy-Gz. For example, if the position (u, v) at the previous time t−1 in the image of the tracking target object changes to the position (u ′, v ′) at the current time t, this object will be in real space (observation (Target space) is estimated to have moved from the position (gx, gy, gz) at the previous time t-1 to the position (gx ', gy', gz ') at the current time t. The change from the previous time t−1 can be acquired.

  Here, the time to be used is a time that is set every time the unit time elapses with the unit time being 1, and the time that is one time before the time t is t−1. Also, the coordinate conversion from the image coordinate system to the world coordinate system as described above can be determined by setting external parameters related to the installation position and shooting direction of each camera 2 in advance by calibration. is there. Even when images are acquired from each of the plurality of cameras 2, these images can be integrated to construct one image space, and an image coordinate system can be applied to this image space.

  As described above, in the present embodiment, the target tracking device 1 is based on the image information (position information in the image coordinate system uv) acquired every moment, and the position information in the real space (world coordinate system Gx) of the tracking target object. (Location information in -Gy-Gz) can be estimated.

[Device configuration, target tracking method]
FIG. 2 is a functional block diagram showing a functional configuration in an embodiment of the object tracking apparatus according to the present invention.

  According to FIG. 2, the target tracking device 1 includes a communication interface 101 that can be connected to one or a plurality of cameras 2, an image storage unit 102, a mask processing image storage unit 103, an identification model storage unit 104, It has a target information storage unit 105 and a processor memory. Here, the processor memory realizes the object tracking function by executing a program that causes the computer of the object tracking apparatus 1 to function.

  Furthermore, the processor memory includes a mask processing unit 111, a tracking area determination unit 112, a discriminator 11, an image output unit 113 including a learning unit 113a and a discrimination unit 113b, and a target position determination unit 114 as functional components. And a tracking target management unit 115 and a communication control unit 121. Note that the processing flow shown by connecting the functional components of the target tracking device 1 in FIG. 2 with arrows is understood as an embodiment of the target tracking method according to the present invention.

  Similarly, in FIG. 2, the camera 2 is a visible light, near-infrared, or infrared imaging device including a solid-state imaging device such as a CCD image sensor or a CMOS image sensor. As described above, a depth camera can be used as the camera 2. Further, the camera 2 or a camera control device (not shown) generates captured image data including an image of an object captured by the camera 2 and transmits the data to the target tracking device 1 in time series or batch. It has a function. It is also preferable that the camera 2 is movable and can change the installation position, shooting direction, and height, and has a function of receiving and processing a control signal for this change.

  The communication interface 101 receives captured image data that is a time-series image group from the camera 2 or the camera control device via a communication network. Control of transmission / reception and communication data processing using the communication interface 101 is performed by the communication control unit 121, and acquired captured image data (image file) is stored in the image storage unit 102. Here, the captured image data may be obtained by being called in chronological order from the camera 2 or the camera control device, and sequentially obtained images captured at regular time intervals in real time. Also good.

  The mask processing unit 111 is an object included in the time-series image group read from the image storage unit 102 or a processing target region in the image region in the image other than one object that is a tracking processing target. A mask process (pixel change process) is performed for changing to a pixel pattern in which the above feature is eliminated or reduced.

  Here, the processing target area to be subjected to the mask process is determined based on the determined position (a) at a past time point or (b) as a correct answer for these other objects. For example, the processing target area at time t may be the correct image area of the other object at time t−1. Incidentally, the mask processing unit 111 can acquire information on the tracking target region determined by the image (or image region) at the past time from the target position determination unit 114 described later.

  In any case, the mask process of the present embodiment is a process for preventing the appearance of other objects from appearing at all or clearly in the image or image area (image patch) to be input to the discriminator 11. ing. Note that it is also preferable that the mask processed image or the image region processed by the mask processing unit 111 is stored in the mask processed image storage unit 103 and is read and used as appropriate.

Here, as the mask processing performed in the mask processing unit 111, the following three that have already been briefly described, that is, (A1) processing for changing the processing target region to a region having a predetermined pixel pattern,
(A2) Processing to change the processing target region to a region having a pixel pattern determined based on the image region that is the background of the tracking target, and (A3) Processing to blur the appearance of other tracking target to the processing target region Can be mentioned.

  Among these, as a specific example of the mask processing (A1), the processing target region may be painted with a single color such as black. By the way, when the discriminator 11 is to be learned (as teacher data) using a mask-processed image or image area subjected to such mask processing, a portion of a predetermined pixel pattern (such as a monochromatic pattern such as black) is to be tracked It can also be understood that negative learning that the object is not an area is actively performed. In other words, it can be said that learning is performed that the area of the tracking target object exists in the image area excluding the portion of the predetermined pixel pattern.

  As a specific example of the mask process (A2), a background model in an image may be learned using a known background modeling technique, and the processing target area may be filled with the learned background color. Here, as the background model, for example, the color of each pixel (pixel) may be an average color at a plurality of times, or the background color distribution in each pixel is modeled by a mixed Gaussian distribution. It is also possible to use it.

  The background modeling method using this Gaussian distribution is, for example, non-patent literature: P. KadewTraKuPong and R. Bowden, "An improved adaptive background mixture model for real-time tracking with shadow detection", Video-Based Surveillance Systems, Computer Vision and Distributed Processing, 2002, 134-144, and non-patent literature: T. Bouwmans, F. El Baf, B. Vachon, "Background Modeling using Mixture of Gaussians for Foreground Detection-A Survey", Recent Patents on Computer Science 1, 2008, pages 219-237

  Furthermore, in the mask process of (A3) above, it may be blurred with a color calculated from the surrounding colors of other tracking objects, and the mask process target area can be blurred by the same method as the image smoothing process. Specifically, a smoothing process may be performed by setting a predetermined kernel and performing a convolution operation with the kernel on the mask processing target area.

  FIG. 3 is a schematic diagram showing an embodiment of a pixel change process (mask process) according to the present invention.

  As shown in FIG. 3A, two images of the current time t as the current time in the tracking process and the previous time t ′ (= t−1) are obtained from the acquired time-series image group. The mask processing when performing tracking using the method will be described. A specific method for inputting and tracking such two images to the discriminator 11 will be described later in detail with reference to FIG. As will be described later in detail, tracking with mask processing is performed using three or four images including images at time t ″ (t ″ <t ′) past time t ′. It is also possible to do.

In the present embodiment, from two images at time t and time t ′, a tracking target area (c _u ^t , c _v ^t , The tracking process is performed by estimating w ^t , h ^t ). Here, c _u ^t and c _v ^t are the u coordinate and v coordinate of the center of the tracking target area (center of the object i) in the image coordinate system, respectively, and w ^t and ^ht are the width of the tracking target area, respectively. And height.

First, as shown in FIG. 3B, in each image at time t and time t ′ (= t−1), the tracking target region (c _u ^t ′, c) of an object other than the object i at time t ′. _{Based on v} ^t ′, w ^t ′, h ^t ′), a mask process target area that is an image area to be masked is determined. For example, the mask processing target region can be a region including a tracking target region of an object other than the object i (c _u ^t ', c _v ^t ', A · w ^t ', B · h ^t ').

  Here, if A = B = 1, the mask processing target area is the tracking target area itself of an object other than the object i. The size (area) of the mask processing target region is a design matter that can greatly affect the tracking accuracy. For example, it is also preferable to set a larger area in a range where the probability of overlapping with the tracking target area of the object i is equal to or less than a predetermined value.

  Incidentally, as described above, in the embodiment using three or more images as described above, other than the object i at one time (for example, the latest time except the current time t, the intermediate time, or the oldest time). Based on the tracking target area of the object, the mask processing target area can be determined. In any case, the mask processing unit 111 (FIG. 2) performs the mask processing (pixel change processing) as described above on the mask processing target area determined as described above.

  Next, as shown in FIG. 3B, a reference image area is set for each image at time t and time t ′ (= t−1) on which the mask processing is performed, and this reference image is set. The cut target area is determined from the area. The image portion of the cut target area becomes an image area (image patch) that is cut out from each image and input to the discriminator 11 later.

Here, when the reference image area is a tracking target area (c _x ^t ′, c _y ^t ′, w ^t ′, h ^t ′) at time t ′ (= t−1), the clipping target area is, for example, (c _x ^t ', c _y ^t ', C · w ^t ', D · h ^t '). Here, if C = D = 2, the cut target region is a region having an area four times that including the tracking target region.

  Incidentally, in the embodiment using three or more images as described above, similarly to the object other than the object i at one time (for example, the newest time, the intermediate time, or the oldest time excluding the current time t). The cut target area can be determined based on the tracking target area. In any case, next, an image portion (image patch) is cut out from the determined cut target area, the image patch is input to the discriminator 11, and the tracking target of the object i at time t is output from the output of the discriminator 11. The area is determined or the classifier 11 is learned.

  Naturally, the reference image area serving as a reference for the mask processing target area and the cut target area is not limited to the tracking target area at a time before the current time t. That is, the reference image area may be the same (in the same coordinate range) image area in the images at each time as described above, but may be set to be different image areas at each time. For example, in the image at each time, the tracking target area at the previous time (for example, the previous time) may be set as the reference image area, and the image area at the time predicted from the trajectory information of the object position is set as the reference image area. It is also possible to do.

  FIG. 4 is a schematic diagram showing another embodiment of pixel change processing (mask processing) according to the present invention.

  In the present embodiment, as shown in FIG. 4A, in the mask processing image at one time, the tracking target region of the object i that is one tracking target and the mask processing target region of another object are superimposed. If so, take measures to maintain or improve the identification accuracy. In fact, in the case of such superimposition, a part of the image information related to the object i is replaced with the pixel information related to the mask processing, so the discriminator 11 can input such a mask processed image (or image). From the patch), a part of correct information for identification or learning cannot be received, or conversely, incorrect information is accepted.

  Therefore, in the present embodiment, as shown in FIG. 4B, when at least one of the one or more mask processing target areas has a superposed area superimposed on the tracking target area of the object i, this ( Mask processing (pixel change processing) is not performed on one or more overlapping regions. Thereby, it is possible to cause the discriminator 11 to receive all correct information in the tracking target area.

  Further, as a change mode, as shown in FIG. 4C, when at least one of the one or more mask processing target areas has a superimposition area superimposed on the tracking target area of the object i, the object i It is also preferable to change the tracking target area to an area excluding this (one or more) overlapping areas. Incidentally, such a region change process is performed by the tracking region determination unit 112 (FIG. 2).

  Here, the cut target area for determining the image patch input to the discriminator 11 is set based on the changed tracking target area. Further, when an image patch from which such a superposed region is excluded is input to the discriminator 11, the entire tracking target region of the object i is changed from the region estimated based on the output of the discriminator 11. You may decide in the form of returning.

  FIG. 5 is a schematic diagram showing still another embodiment of the pixel change process (mask process) according to the present invention.

  In the present embodiment, particularly when three or more images are used, the mask processing target region is not greatly deviated from the actual image position of an object other than the object i when the image times are separated from each other by a predetermined amount or more. To take action.

  For example, as shown in FIG. 5A, from the time-series image group, an image at the current time t and an image at the time t ″ (t ″ <t) separated from the current time t by a predetermined distance or more. A case will be described in which two or more image groups including the above are extracted and mask processing is performed. In this case, in the image at time t '', for example, masking processing may be performed using the tracking target region of an object other than the object i as a mask processing target region, whereas in the image at time t, It is also preferable to perform mask processing on the approximate position range of these other objects.

Here, as an approximate position range of other objects at time t, for example, as shown in FIG. 5B, an average velocity vector v (t '') around time t '' in each other object. And the estimated movement vector r from time t ″ to time t is expressed by the following equation (1) r = v (t ″) · (t ″ −t)
The area to which the tracking target area of each other object at time t '' is translated in parallel by the estimated movement vector r of each other object is determined as the mask processing target area of each other object in the image at time t. It is good.

  As a change mode, it is also possible to predict the position at time t based on the trajectory information from time t '' to time t-1 for the position of each other object, and determine the mask processing target area. It is.

  Further, as a change mode, when the mask processing unit 111 (FIG. 2) learns the discriminator 11, the mask processing is performed on the shift processing target region whose position is shifted from the mask processing target region determined as described above. May be implemented. Here, the position shift may be a shift by a random shift amount within a predetermined range in a random direction, or may be a shift by a predetermined shift amount in a predetermined direction set in advance. The discriminator 11 learns a mask processing image (image patch) related to such a shift processing target region in advance, so that the masked region should be actually masked during actual detection (identification). More appropriate detection (identification) can be performed on a mask-processed image (image patch) that is not likely to be displaced from the position of the object to some extent.

  Returning to FIG. 2, the image output unit 113 outputs, to the discriminator 11, one or more images or image areas that have been subjected to the mask processing (pixel change processing) as described above. Here, the learning unit 113a of the image output unit 113 outputs a mask processed image or a mask processed image region (image patch) to the classifier 11 for learning one object i that is a tracking target.

  On the other hand, the identification unit 113b of the image output unit 113 outputs the mask processed image or the mask processed image patch to the classifier for identifying the object i. Here, in performing multiple object tracking (multi-tracking), mask processing is performed in which the objects to be tracked are handled as objects i one by one in order, and the mask processing image (image patch) is used for the classifier 11. On the other hand, identification of each object (determination of the image area position of the object) is sequentially performed.

  Incidentally, the input of the mask processed image (mask processed image patch) may be performed only at the time of identification by the classifier 11. In this case, learning of the discriminator 11 is performed by inputting an image (image patch) that has not been subjected to mask processing. Further, the mask processed image (mask processed image patch) may be input only during learning of the classifier 11, but the mask processed image (mask processed image patch) is input in both learning and identification. It is also preferable. That is, it is also preferable to perform discrimination on an image (image patch) that has been subjected to mask processing using the discriminator 11 that has been learned from the mask processing image (mask processing image patch).

  Next, two embodiments of the discriminator 11 (FIG. 2) of the non-online learning type and the online learning type will be described with reference to FIGS. 6 and 7, respectively.

  FIG. 6 is a schematic diagram showing an embodiment of a discriminator according to the present invention.

As shown in FIG. 6, the discriminator 11 is, in this embodiment,
(A) Convolutional Layers (Convolutional Layers) as a plurality of first neural networks that input an image or image region (image patch) at each time point and output feature information related to these features;
(B) Fully-connected layers as a second neural network that inputs a plurality of feature information outputted from these convolutional layer parts and outputs information related to the position of one tracking target. Including.

  Here, the convolution layer part of (a) has a function imitating the function of a simple cell in the visual cortex of an animal, and generates a feature map by sliding a kernel (weighting matrix filter) on an image. Execute. With this convolution process, it is possible to extract basic features such as edges and gradients while gradually reducing the resolution of the image, and obtain information on local correlation patterns.

  For example, AlexNet using five convolution layers can be used as the convolution layer portion. In this AlexNet, each convolution layer is paired with a pooling layer, and the convolution process and the pooling process are repeated. Here, the pooling process is a process that mimics the function of complex cells in the visual cortex of animals. The feature map output from the convolution layer (convolution filter response in a certain area) is expressed as a maximum value or an average value. In summary, it is a process of ensuring local translational invariance while reducing adjustment parameters. AlexNet is described in, for example, Krizhevsky, A., Sutskever, I., and Hinton, GE, “Imagenet classification with deep convolutional neural networks”, Advances in Neural Information Processing Systems 25, 2012, pages 1106 to 1114. Yes.

  Furthermore, for example, a neural network described in Non-Patent Document 2 may be employed as the convolution layer portion and the total coupling layer portion. Incidentally, in the present embodiment, the number of convolution layer parts (first neural network) may be two, but in order to achieve higher identification accuracy (tracking accuracy), it is set to three or more as shown in FIG. Is also preferable.

The discriminator 11 performs offline learning in advance before performing the tracking target detection (identification) process at each time point. Specifically,
(A) a set of two or three or more images (or image patches) that fluctuate in time;
(B) The image output unit 113 learns a data set including a four-dimensional vector (CU _C , CV _C , CW, CH) that defines the position and range of the correct tracking target area in each image (or each image patch). Learning is performed by inputting a large amount from the unit 113a.

Here, each CU _C and CV _C, which is the object the center of the u-axis direction of the object center and the v-axis direction of the correct image area. CW and CH are the width and height of the correct image region rectangle, respectively. When the input is an image patch, the coordinate system based on the CU _C value and the CV _C value is the local coordinate system of the image patch. The input image (image patch) is also preferably a mask processed image (image patch) subjected to the mask processing as described above.

  Note that the discrimination model generated by the discriminator 11 that has learned in this way is also preferably stored in the discrimination model storage unit 104 (FIG. 2). For example, a stored identification model may be used by being transplanted to an external classifier.

  Next, in the detection (identification) of a plurality of objects to be tracked, the plurality of convolution layer portions of the discriminator 11 are masked images (images) at time t from the identification unit 113b (FIG. 2) of the image output unit 113. A mask processing image (image patch) group including a patch is input. Here, it is also preferable that each convolution layer unit inputs each of a plurality of times having a time interval set at the time of learning in the same time order (time allocation) as at the time of learning.

  In FIG. 6, the mask processing image (image patch) at the current time t is input to the uppermost convolution layer portion in the drawing, and the time t ′ (t ′ <t) is input to the second convolution layer portion from the top in the drawing. The mask processed image (image patch) at time t ″ (t ″ <t ′) is input to the third convolutional layer portion from the top in the figure. . Here, in the present embodiment, at the time of learning, the uppermost convolutional layer portion in the figure, the second convolutional layer portion from the top, and the third convolutional layer portion from the top respectively have a certain time T and time (T−t + t ') And an image (image patch) or a mask-processed image (image patch) at time (T−t + t ″) are input and learned.

Further, in the detection (identification) of a plurality of objects to be tracked, the discriminator 11 (a plurality of convolution layers) sequentially inputs an image (image patch) subjected to mask processing for each of the plurality of objects as an identification target. Multi-tracking is realized by (and outputting the identification result).

Next, as described above, all the connection layers to which the feature amount output from the convolution layer to which the mask processing image (image patch) is input are finally converted into four-dimensional vectors (U _C , V _C , W, H ) Is output. Here, U _C and V _C are the object center in the u-axis direction and the object center in the v-axis direction of the tracking target region for the object to be identified in the input mask processing image (image patch), respectively. W and H are the width and height of the tracking target area rectangle, respectively. Incidentally, if the input is an image patch, the coordinate system based on the U _C value and V _C value is a local coordinate system of the image patch.

  Further, as shown in FIG. 6, as a modification, the image output unit 113 (FIG. 2) has a convolutional layer unit that performs the learning and identification of the discriminator 11 from the convolutional layer unit (second neural network). Information related to the time point in the image (image patch) corresponding to the output feature information (the time interval Δt between images in FIG. 6) may be output in association with this feature information. Thereby, the discriminator 11 can learn also the information regarding the time of the several image (image patch) input, and based on it, it becomes possible to implement a more accurate detection (identification). In addition, a variety of time can be secured in a plurality of images (image patches) input to the discriminator 11.

  In any case, the discriminator 11 inputs from the image output unit 113 an image (image patch) that has been subjected to mask processing for an object other than the tracking target object at that stage. The accuracy is improved and the occurrence of drift can be suppressed.

  FIG. 7 is a schematic diagram showing another embodiment of the discriminator according to the present invention.

  According to FIG. 7, the discriminator 11 ′ of this embodiment has a configuration in which a support vector machine (SVM) capable of performing machine learning is connected to the output side of a convolutional neural network (CNN) including a convolution layer.

  The discriminator 11 ′ learns the mask processed image area (image patch) input from the image output unit 113 every moment (at the current time t) online, and applies this image patch to the mask processed image patch. A binary determination is made as to whether or not the object shown is a tracking target object. Specifically, while generating and updating the identification boundary surface in the feature space, the signed distance d from the identification boundary surface is calculated as the reliability, and it is determined whether the reliability is equal to or greater than a predetermined threshold. It is. Note that object tracking using such a classifier is, for example, S. Hare, A. Saffari and PHS Torr, “Struck: Structured Output Tracking with Kernels”, Publications of International Conference on Computer Vision (ICCV), 2011. Year, pages 263-270.

  Note that the mask-processed image patch (at the current time t) that is output every moment by the image output unit 113 can be a plurality of image patch candidates that can be the correct tracking target area at the time t. In this case, the discriminator 11 ′ discriminates the correct tracking target area from these candidates.

  As described above, the discriminator 11 ′ inputs an image (image patch) on which mask processing is performed on an object other than the tracking target object at that stage from the image output unit 113. The identification accuracy is improved and the occurrence of drift can be suppressed.

Returning to FIG. 2, the target position determination unit 114 determines the position of the tracking target object at the current time t based on the output from the learned discriminator 11. Specifically, for each of a plurality of objects to be tracked, tracking in the image at time t is finally performed based on the four-dimensional vector (U _C , V _C , W, H) output from the classifier 11. The target area (c _u ^t , c _v ^t , w ^t , h ^t ) is calculated. As a change mode, the target position determination unit 114 may determine the position of the tracking target object at the current time t based on the output from the discriminator 11 ′ (FIG. 7) learned online.

The tracking target management unit 115 uses the information on the tracking target regions (c _u ^t , c _v ^t , w ^t , h ^t ) in each of the plurality of objects that are the tracking targets determined by the target position determination unit 114, For each of a plurality of objects, tracking history information (flow line information) in which positions in the world coordinate system Gx-Gy-Gz (corresponding to real space) are associated with each other is generated and managed. As one application example, this makes it possible to more accurately grasp the flow lines of a large number of store clerks and customers who stay and move in the store.

  The generated tracking history information (flow line information) is also preferably stored in the target information storage unit 105 every time it is generated / updated or as appropriate. Further, the information may be transmitted to the external information processing apparatus 3 via the communication control unit 121 and the communication interface 101.

  As described above in detail, when tracking a plurality of objects as tracking targets, the present invention performs identification by performing “pixel change processing” on other objects other than one object that is a tracking processing target. The characteristics of other objects in the image or image area that are input to the vessel are eliminated or reduced. As a result, this one object is confused with another object by using the image or image region that has been subjected to such processing to learn the classifier and / or to have the classifier perform the classification process. It becomes possible to continue specifying more reliably without doing.

  In other words, by using the “image object mask method” of the present invention described above, even when an apparently similar object exists near the object to be tracked, the drift that has been a problem in the past is generated. It becomes possible to suppress.

  By the way, this "image object mask method" enables both the suppression of memory consumption (calculation cost) increase and the suppression of drift occurrence, especially when applied to multi-tracking processing without online learning. It will be a very important technology.

  In addition, the configuration and method of the present invention include, for example, a monitoring system for monitoring a place where a large number of persons move, stay, and enter and exit, and a person entering, resting, watching / It can be applied to various systems such as event surveys and marketing survey systems for investigating the status of travel.

  In the various embodiments of the present invention described above, various changes, modifications, and omissions in the technical idea and scope of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Target tracking apparatus 101 Communication interface 102 Image storage part 103 Mask process image storage part 104 Identification model storage part 105 Target information storage part 11, 11 'Classifier 111 Mask process part 112 Tracking area | region determination part 113 Image output part 113a Learning part 113b Identification unit 114 Target position determination unit 115 Tracking target management unit 121 Communication control unit 2 Camera 3 Information processing device

Claims (14)

Using a time-series image group that can include a plurality of tracking targets, the tracking target is determined by determining the position of the tracking target in each image based on the output from the discriminator that has input the image or the image area in the image. A device for tracking
A processing target region in an image included in the image group or an image region in the image and determined based on a position determined at a past time or as a correct answer for other tracking targets other than one tracking target Mask processing means for performing a pixel change process for changing to a pixel pattern in which the characteristics of the other tracking target are eliminated or reduced with respect to the target region;
An object tracking device comprising: an image output unit that outputs the image or the image area subjected to the pixel change process to the classifier for learning and / or identification of the one tracking target. .   The mask processing means is a pixel pattern determined based on an image region other than the tracking target in the image or the image region, or a predetermined pixel pattern for at least one processing target region related to the other tracking target. The object tracking apparatus according to claim 1, wherein the object tracking apparatus performs a change to the function.   The said mask processing means implements the process which blurs the appearance of this other tracking object with respect to the process target area | region which concerns on at least one said other tracking object as the said pixel change process. 2. The object tracking device according to 2.   When the processing target area related to the other tracking target has a superimposing area superimposed on the tracking target area including the position determined for the one tracking target, the mask processing means The object tracking device according to claim 1, wherein the pixel changing process is not performed.   When the processing target area related to the other tracking target has a superimposed area superimposed on the tracking target area including the position determined for the one tracking target, the tracking target area related to the one tracking target is 5. The object tracking device according to claim 1, further comprising a tracking target area determining unit that changes to an area excluding the superimposition area. 6. The image output means outputs the image or the image region subjected to the pixel change processing at at least one time point to the learned classifier,
2. The target tracking device further includes target position determining means for determining a position of the one tracking target at the one time point based on an output from the learned classifier. 6. The object tracking device according to any one of 1 to 5.   The image output means includes the image or the image area subjected to the pixel change process at one time point, and the image subjected to the pixel change process at at least one time point before the one time point. The object tracking device according to claim 6, wherein the image region is output to the learned classifier.   The discriminator receives a plurality of first neural networks that input the image or the image region at each time point and outputs feature information relating to the feature of the image or the image region, and outputs from the plurality of first neural networks The object tracking device according to claim 6, further comprising: a second neural network that inputs the plurality of pieces of the feature information and outputs information related to the position of the one tracking target.   9. The image output means outputs the information corresponding to the feature information or the time point of the image region to the second neural network in association with the feature information. Object tracking device as described in.   The image output means, when learning the discriminator, at least the image or the image region that has been subjected to the pixel change process, and information on the position of the tracking target region as a correct answer in the image or the image region The object tracking device according to claim 1, wherein a set of the following is output to the discriminator.   The said mask processing means implements the said pixel change process with respect to the shift | offset | difference process target area | region shifted in position from the determined said process target area, when learning the said discriminator. The object tracking device according to any one of the above. The image output means outputs at least the image or the image region at one time point subjected to the pixel change processing to the discriminator that performs online learning,
The target tracking device further includes target position determining means for determining the position of the one tracking target at the one time point based on an output from the discriminator that is learning online. The object tracking device according to any one of claims 1 to 5. Using a time-series image group that can include a plurality of tracking targets, the tracking target is determined by determining the position of the tracking target in each image based on the output from the discriminator that has input the image or the image area in the image. A program for operating a computer mounted on a device for tracking
A processing target region in an image included in the image group or an image region in the image and determined based on a position determined at a past time or as a correct answer for other tracking targets other than one tracking target Mask processing means for performing a pixel change process for changing to a pixel pattern in which the characteristics of the other tracking target are eliminated or reduced with respect to the target region;
A computer is caused to function as an image output means for outputting the image or the image region subjected to the pixel change processing to the discriminator for learning and / or identification of the one tracking target. Target tracking program. Using a time-series image group that can include a plurality of tracking targets, the tracking target is determined by determining the position of the tracking target in each image based on the output from the discriminator that has input the image or the image area in the image. An object tracking method in a computer mounted on a device for tracking
A processing target region in an image included in the image group or an image region in the image and determined based on a position determined at a past time or as a correct answer for other tracking targets other than one tracking target Performing a pixel change process for changing to a pixel pattern in which the characteristics of the other tracking target are eliminated or reduced with respect to the target region;
Outputting the image or the image region subjected to the pixel changing process to the discriminator for learning and / or identification of the one tracking target.

Images