JP2021117635A - Object tracking device and object tracking method - Google Patents

Abstract

To precisely track an object.SOLUTION: An object tracking device 1 includes: an object detection unit 122 that detects an object from a plurality of captured images of a prescribed area captured at each of a plurality of times; a motion feature amount extraction unit 124 that extracts a motion feature amount of the object on the basis of partial images, of the captured images, corresponding to the detected object; and an association unit 125 that associates each of a plurality of objects detected from each of the plurality of captured images captured at different image capturing times on the basis of the motion feature amount. The motion feature amount extraction unit 124 re-extracts, for each of the associated objects, a motion feature amount on the basis of a partial image corresponding to the object and a partial image corresponding to another object associated with the object. The association unit 125 re-associates objects detected from each of the plurality of captured images captured at different image capturing times, on the basis of the re-extracted motion feature amount.SELECTED DRAWING: Figure 3

Description

The present invention relates to an object tracking device and an object tracking method.

Conventionally, an object such as a person appearing in a captured image captured by an imaging device has been tracked. For example, in Non-Patent Document 1, an object is tracked by using a position feature amount indicating a feature related to the position of the object, a visible feature amount indicating the appearance of the object, and a behavioral feature amount indicating the behavioral feature of the object. Is disclosed.

Gurkirt Singh, Suman Saha, Michael Sapienza, Philip H. S. Torr, and Fabio Cuzzolin, "Online Real-time Multiple Spatiotemporal Action Localization and Prediction," IEEE International Conference on Computer Vision (ICCV), pp. 3637-3646, 2017

Non-Patent Document 1 does not consider the relationship between the object reflected in the captured image and the object reflected in another captured image when the behavioral feature amount is extracted from the captured image. However, the behavior is found as a result of the movement, and it is difficult to accurately extract the behavioral features from one captured image. Therefore, Non-Patent Document 1 has a problem that the tracking accuracy for objects having similar positions and appearances is lowered.

Therefore, the present invention has been made in view of these points, and an object of the present invention is to provide an object tracking device and an object tracking method capable of accurately tracking an object.

The object tracking device according to the first aspect of the present invention includes an acquisition unit that acquires a plurality of captured images captured by an imaging device that captures a predetermined area at a plurality of time of three or more, and the acquisition unit. Based on an object detection unit that detects an object reflected in the captured image from each of the plurality of captured images acquired by the object, and a partial image of the captured image corresponding to the object detected by the object detection unit. The behavioral feature amount extraction unit that extracts the behavioral feature amount indicating the behavioral feature amount of the above, and the plurality of captured images having different imaging times based on the behavioral feature amount extracted by the behavioral feature amount extraction unit, respectively. The object detection unit includes an association unit that associates the objects detected by the object detection unit, and the behavior feature amount extraction unit corresponds to the object for each of the plurality of objects associated with the association unit. The behavioral feature amount is re-extracted based on the partial image to be performed and the partial image corresponding to each of one or more other objects associated with the object, and the association part is generated by the behavior feature amount extraction unit. Based on the re-extracted behavioral feature amount, the object detected by the object detection unit is associated again from each of the plurality of captured images having different captured times.

The acquisition unit acquires a plurality of four or more captured images captured by the imaging device at each of the four or more time periods, and the object tracking device reconstructs the behavior feature amount by the behavior feature amount extraction unit. An execution control unit that tracks the objects may be further provided by alternately and repeatedly executing the extraction and the reassociation of the objects by the association unit until a predetermined condition is satisfied.

As a result of associating the objects by the association unit, the execution control unit has a ratio of the number of objects associated before the association to the number of objects associated after the association. The re-extraction of the behavioral feature amount by the behavioral feature amount extraction unit and the association of the objects by the association unit may be alternately and repeatedly executed until the ratio becomes a predetermined ratio or more.

The behavior feature extraction unit determines the behavior feature of the object based on the behavior tendency indicated by the behavior feature corresponding to the object for each of the plurality of objects associated with the association. You may vary the number of other objects used for re-extraction.

The object detection unit detects the object by specifying an object position indicating the position of the object reflected in the captured image, and the action feature amount extraction unit corresponds to the object position specified by the object detection unit. The behavioral feature amount may be extracted based on the partial image.

The association unit may associate the object detected by the object detection unit from each of the plurality of captured images having different captured times, based on the object position specified by the object detection unit. ..

The object tracking device further includes a visible feature amount extracting unit that extracts a visible feature amount indicating the appearance of the object in the captured image detected by the object detection unit, and the associating unit is the visible feature amount extracting unit. Based on the visible feature amount extracted by the object, the object detected by the object detection unit may be associated with each of the captured images having different captured times.

The behavioral feature amount extraction unit captures the partial image of the captured image corresponding to the object detected by the object detection unit and an image captured at a time before or after the time at which the captured image is captured. The behavioral feature amount may be extracted based on the difference between the image and the partial image.

The behavioral feature amount extraction unit is a first partial image that is a partial image including the object detected by the object detection unit, and the partial image that includes the object and has a larger display area than the first partial image. The behavioral feature amount may be extracted based on the second partial image.

The behavior feature extraction unit is one or more of the above, in which the partial image corresponding to the object is associated with the object in a neural network that outputs the behavior feature in response to the input of the partial image. The partial image corresponding to the other object may be input, and the behavioral feature of the object may be re-extracted based on the plurality of behavioral features output from the neural network.

The behavior feature extraction unit inputs the partial image corresponding to the object into the neural network that outputs the behavior feature in response to the input of the partial image, and obtains the feature indicated by the intermediate layer in the neural network. The behavioral feature amount of the object may be extracted based on the acquired feature amount.

The behavior feature extraction unit is one or more of the above, in which the partial image corresponding to the object is associated with the object in a neural network that outputs the behavior feature in response to the input of the partial image. The partial image corresponding to the other object may be input, the feature amount indicated by the intermediate layer in the neural network may be acquired, and the behavioral feature amount of the object may be re-extracted based on the acquired feature amount. ..

The object tracking method according to the second aspect of the present invention includes a step of acquiring three or more captured images captured by an imaging device that captures a predetermined area at a plurality of times of three or more, which is executed by a computer. , A feature amount of the behavior of the object based on the step of detecting the object reflected in the captured image from each of the acquired plurality of captured images and the partial image of the captured image corresponding to the detected object. The step of extracting the behavioral feature amount indicating the above, and the step of associating the object detected from each of the plurality of captured images having different captured times based on the extracted behavioral feature amount, and the association. For each of the plurality of objects performed, the behavioral feature amount is regenerated based on the partial image corresponding to the object and the partial image corresponding to each of one or more other objects associated with the object. It has a step of extracting and a step of reassociating the objects detected from each of the plurality of captured images having different captured times based on the re-extracted behavioral feature amount.

According to the present invention, there is an effect that an object can be tracked with high accuracy.

It is a figure which shows the outline of the object tracking apparatus which concerns on this embodiment. It is a figure which shows the example of the association of an object. It is a figure which shows the structure of the object tracking apparatus which concerns on this embodiment. It is a figure which shows the extraction example of the behavior feature amount information when the object is not associated. It is a figure which shows the extraction example of the behavior feature amount information in the case where the object is associated. It is a flowchart which shows the flow of processing in the object tracking apparatus which concerns on this embodiment.

[Overview of Object Tracking Device 1]
FIG. 1 is a diagram showing an outline of the object tracking device 1 according to the present embodiment. The object tracking device 1 is a device that tracks objects by associating one or more objects that appear in a plurality of captured images captured by the imaging device 2 that images a predetermined area such as in a store. Here, the object is, for example, a clerk or a customer who acts in the store.

The object tracking device 1 acquires a plurality of time-series captured images captured by the image capturing device 2 ((1) in FIG. 1). The object tracking device 1 detects an object reflected in the captured image from each of the acquired plurality of captured images ((2) in FIG. 1).

The object tracking device 1 extracts a behavioral feature amount indicating the behavioral feature amount of the detected object for each of the plurality of captured images ((3) in FIG. 1). The object tracking device 1 associates the objects detected from each of the plurality of captured images based on the extracted behavioral features ((4) in FIG. 1).

The object tracking device 1 re-extracts the behavioral features of each of the plurality of objects based on the captured image corresponding to the object and another captured image in which the associated object is displayed. Since the objects associated with the plurality of captured images can extract the behavioral features based on the behaviors of the objects shown by the plurality of captured images, the accuracy of the behavioral features is higher than before the association is performed.

The object tracking device 1 associates the objects detected from each of the plurality of captured images based on the re-extracted behavioral features. Due to the increased accuracy of behavioral features, objects that have not been associated with each other will be associated with each other.

FIG. 2 is a diagram showing an example of associating objects. In FIGS. 2A to 2D, the horizontal axis represents the imaging time and the vertical axis represents the position of the object. Further, in FIGS. 2A to 2D, the mark M indicates the detected object. In FIGS. 2A to 2D, the marks M shown at the same time indicate the same object detected by the object tracking device 1. In time t _3, but not present mark M, which in the same time, for example because of the shielding, etc., indicate that it could not detect objects. In addition, the alphabet shown in the mark M indicates the class information indicating the behavior class indicated by the behavior feature amount.

FIG. 2A shows a state before the object tracking device 1 associates objects. FIG. 2B shows a state in which the object tracking device 1 first makes an association.

FIG. 2C shows a state in which the object tracking device 1 re-extracts the behavioral features of the object after the association is performed for the first time. Association results of the re-extraction action feature quantity by objects made were made, as shown in FIG. 2 (c), object class information at time t ₂ is changed from "b" to "a" with, class information of the object at time t ₄ it can be confirmed that has changed from "c" to "a".

FIG. 2D shows a state in which the object is associated again after the behavioral feature amount of the object is re-extracted as shown in FIG. 2C. As shown in FIG. 2 (d), the result of action feature quantity of an object in the object and the time t ₄ at time t ₂ is re-extracted, that an object in the object and the time t ₄ at time t ₂ is associated with You can check. Thus, even if it can not detect objects at time t _3, the action feature quantity at a time t _2, t ₄ before and after the time t ₃ by the re-extraction of the object at time t _2, t ₄ class information is corrected to the same thing, it is possible to associate an object at time t _2, t _4.

Returning to FIG. 1, the object tracking device 1 repeatedly executes the process (3) and the process (4) of FIG. The object tracking device 1 can improve the accuracy of the object association by alternately re-extracting the behavioral feature amount and associating the objects. As a result, the object can be tracked with high accuracy. ..
Hereinafter, the configuration of the object tracking device 1 will be described in detail.

[Configuration of object tracking device 1]
FIG. 3 is a diagram showing a configuration of the object tracking device 1 according to the present embodiment. As shown in FIG. 3, the object tracking device 1 includes a storage unit 11 and a control unit 12.

The storage unit 11 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The storage unit 11 stores a program executed by the control unit 12. For example, the storage unit 11 is an object tracking program that causes the control unit 12 to function as an acquisition unit 121, an object detection unit 122, a visible feature amount extraction unit 123, an action feature amount extraction unit 124, an association unit 125, and an execution control unit 126. I remember.

The control unit 12 is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). By executing the object tracking program stored in the storage unit 11, the control unit 12 executes the acquisition unit 121, the object detection unit 122, the visible feature amount extraction unit 123, the action feature amount extraction unit 124, the association unit 125, and the execution unit. It functions as a control unit 126.

[Acquisition of captured image]
The acquisition unit 121 acquires a plurality of captured images of 4 or more captured by the imaging device 2 that captures a predetermined area at each of the plurality of times of 4 or more. Here, it is assumed that the imaging device 2 images the same imaging range at each of a plurality of times.

[Object detection]
The object detection unit 122 detects an object reflected in the captured image from each of the plurality of captured images acquired by the acquisition unit 121. Specifically, the object detection unit 122 detects an object by specifying an object position indicating the position of the object reflected in the captured image.

For example, the object detection unit 122 is, for example, an object detector. When the captured image acquired by the acquisition unit 121 is input, the object detection unit 122 has the position feature amount information X ^loc indicating the feature amount of the position of the object with respect to the captured image, and the object exists at the position. The object is detected by outputting the _{cost C date} indicating the certainty. As the object detector, for example, SSD (Single Shot Multibox Detector) is used.

The position feature amount information X ^loc is indicated by, for example, a combination of four variables (x, y, w, h). x is the position on the X-axis direction (horizontal direction) in the captured image, y is the position on the Y-axis direction (vertical direction) orthogonal to the X-axis in the captured image, and w is the length (width) of the object in the X-axis direction. ) And H indicate the length (height) of the object in the Y-axis direction. The position feature amount information X ^loc identifies a bounding box which is a rectangular area surrounding the detected object. The object detection unit 122 allocates an index i to each of the objects detected in each of the plurality of captured images, and identifies the _{position feature amount information X i} ^loc and the cost C _{det (i) corresponding to each object i.}

[Extraction of visible features]
The visible feature amount extraction unit 123 extracts a visible feature amount indicating how the object detected by the object detection unit 122 looks in the captured image. For example, the visible feature amount extraction unit 123 specifies the position and size of the bounding box in the captured image based on ^{the position feature amount information X loc detected by the object detection unit 122 from the captured image.} The visible feature amount extraction unit 123 extracts a partial image surrounded by the specified bounding box as a partial image showing an object from the captured image.

The visible feature amount extraction unit 123 outputs the extracted partial image to the visible feature amount output program that outputs the visible feature amount ^{information X app} , which is a multidimensional vector indicating the feature amount of the appearance of the object with respect to the input of the image. The visible feature amount of the object is extracted by inputting and acquiring the visible feature amount information X ^{app output from the program.} The visible feature output program is, for example, a deep neural network program, and for example, WRNs (Wide Residual Networks) are used.

Feature amount extraction unit 123 visible, enter the feature quantity output program looks a partial image corresponding to each object i to the object detection unit 122 has detected the appearance feature amount information X _i ^app corresponding to each object i from the program get.

[Extraction of behavioral features]
The behavioral feature extraction unit 124 extracts behavioral features indicating the behavioral features of each of the plurality of objects based on the partial images of the captured images corresponding to the plurality of objects detected by the object detection unit 122. .. Action feature quantity extraction unit 124, based on the partial image corresponding to the position feature quantity information X _i ^loc indicating the object positions of the plurality of objects i the object detection unit 122 has detected, corresponding to each of the plurality of objects i action Extract features. The behavioral feature is a multidimensional vector, and the number of dimensions corresponds to, for example, the number of behavioral classes.

[Extraction of behavioral features when no association is made]
The behavior feature extraction unit 124 is a multidimensional vector indicating the behavior feature of the object i detected by the object detection unit 122, as shown below, when the objects are not associated with each other by the association unit 125. Behavioral feature amount information X _i ^paf is extracted. Figure 4 is a diagram showing an example of extraction of a behavioral characteristic quantity information X _i ^paf when the association object is not performed.

Action feature quantity extraction unit 124 extracts the action feature amount information X _i ^paf-C based on the color of the partial images of the object i, and the partial images, from the front and back of the captured image of the captured image in which the partial image is extracted extracting the action feature amount information X _i ^paf-F based on the difference between the extracted partial image. Then, the behavior characteristic amount extraction unit 124, by integrating the behavior characteristic amount information X _i ^paf-C based on the color of the partial images, a behavior characteristic amount information based on the difference of the partial image X _i ^paf-F, Object The behavioral feature amount information X _i ^paf of i is extracted.

First, the behavior characteristic amount extraction unit 124, based on the positional characteristic amount information X _i ^loc object i to the object detection unit 122 detects, from the captured image in which the object is detected, as a first partial image containing the object The local cutout image Im _{L of the above and the global cutout image Im G} as the second partial image which is a partial image including the object and having a display area larger than that of the first partial image are extracted.

Local clipped images Im _L is, for example, an image to be surrounded by a bounding box indicating the position feature quantity information _X ^{i loc} object i, global cut image Im _G includes a local clipped images Im _L, local switching This is an image having a display area several times larger than that of the output image Im _L. Then, the behavior characteristic amount extraction unit 124 extracts the action feature amount information _X ^{i paf} based on the extracted local cut image Im _L and global clipped images Im _G. By doing so, the object tracking device 1 can extract the behavioral feature amount with higher accuracy in consideration of other objects existing around the object and the information of the object.

In the present embodiment, the object tracking apparatus 1 has been decided to extract behavioral feature amount information X _i ^paf on the basis of the local cut image Im _L and the global clipped images Im _G, not limited thereto, local clipped images Im _L and the global cut image Im or behavioral characteristic quantity information X _i ^paf based on one of the _G may be extracted.

In the example shown in FIG. 4, the behavioral feature amount extraction unit 124 _{obtains the color information indicated by the local cutout image Im L} and the color information indicated by the global cutout image Im _G into deep layers such as ResNet (Residual Networks), respectively. enter the base model program of the neural network, by combining the action feature amount information output from the base model program, it extracts the action feature amount information based on a color X _i ^paf-C.

_{In addition, the behavioral feature amount extraction unit 124 captured the partial image (local cutout image Im L} and global cutout image Im _G ) of the object i extracted from the captured image in which the object i was detected, and the captured image. based on the difference between the partial image of the previous time or after the time the captured image captured at the time, extracts the action feature amount information X _i ^paf-F.

For example, action feature quantity extraction unit 124 extracts from the captured image of the imaging time _{t i} the object i is detected, the local cut image Im _L and global cut image Im _G on the basis of the positional feature amount information _X ^{i loc} do. Further, the behavior characteristic amount extraction unit 124, from the immediately preceding imaging time _{t i-1} of the captured image capturing time _{t i} the object i is detected based on the positional characteristic amount information _X ^{i loc} local cut image _{Im Lb} And the global cutout image Im _Gb is extracted. Then, the behavior characteristic amount extraction unit 124, the difference D _L of the extracted two local clipped _images, and based on the difference D _G of the two global clipped images, extracts a behavior feature amount information X _i ^paf-F ..

In the example shown in FIG. 4, the behavior characteristic amount extraction unit 124, the difference D _L of the local clipped images, and the difference D _G global clipped images, respectively, input to the base model program deep neural network such ResNet and extracts the action feature amount information X _i ^paf-F by combining the action feature amount information output from the base model program. It is assumed that the base model program has been learned in advance.

Then, the behavior characteristic amount extraction unit 124 by integrating the extracted and behavioral characteristic quantity information _X ^{i paf-C} and action feature amount information _X ^{i paf-F,} extracting action feature amount information _X ^{i paf} object i do. For example, action feature quantity extraction unit 124, the average value of the extracted and behavioral characteristic quantity information _X ^{i paf-C} and behavioral characteristic quantity information _X ^{i paf-F} and action feature quantity information _X ^{i paf} object i.

The behavioral feature amount extraction unit 124 uses the behavioral feature amount information Xi ^paf-C and the behavioral feature amount information X _i ^paf-C _{as another method of integrating the behavioral feature amount information X i} paf-C and the behavioral feature amount information X _i ^paf-F. The X _i ^paf-F may be compared for each element, and the vector composed of the larger value may be ^used _{as the behavioral feature amount information X i paf of the object i.} For example, with X _i ^paf-C = (0.1, 0.2, ^{0.5, 0.6} _{) and X i} paf-F = (0.9, 0.6, 0.2, 0.9). In some cases, X _i paf = ( ^{0.9, 0.6, 0.5, 0.9).} Hereinafter, this extraction method will be referred to as a maximum element extraction method.

Further, the behavior characteristic amount extraction unit 124, extracting the action feature amount information X _i ^paf, among vector value indicated by the action feature amount information X _i ^paf, the class information indicating the action class corresponding to the highest vector value Extract.

Note that action feature quantity extraction unit 124 has been extracted action feature amount information X _i ^paf as shown in FIG. 4, not limited to this. Action feature quantity extraction unit 124 may extract the action feature amount information X _i ^paf using a program of deep neural networks, such as TSN (Temporal Segment Networks).

Also, if the association between an object is not performed, the average value of the extracted and behavioral characteristic quantity information X _i ^paf-C and behavioral characteristic quantity information X _i ^paf-F is, necessarily reflect the behavioral features of the object Rather, it is considered preferable to give ambiguity to the behavioral features. Therefore, the behavior characteristic amount extraction unit 124, extracts acquires the feature quantity indicated by the intermediate layer in each neural network, what was averaging such as a plurality of feature amount acquired, as the action feature amount information X _i ^paf You may. By doing so, the object tracking apparatus 1, when the association of the object has not been performed, it is possible to have ambiguity in the behavior characteristic amount information X _i ^paf.

[Extraction of behavioral features when association is performed]
When the objects are associated with each other by the association unit 125, the behavior feature amount extraction unit 124 has a partial image corresponding to the object i and the partial image of each of the plurality of objects i associated with the association unit 125. The behavioral features are re-extracted based on the partial images corresponding to each of the one or more other objects associated with the object i. Here, the behavioral feature amount extraction unit 124 captures images taken within a predetermined time (predetermined number of frames) from the imaging time of the captured image in which the object i is projected, among one or more other objects associated with the object i. The behavioral features shall be re-extracted based on the partial images of other objects detected from the image. In the following description, the reference range of another object used for extracting the behavioral feature amount of the object i, which is determined by a predetermined number of frames, is referred to as a window size.

Figure 5 is a diagram showing an example of extraction of a behavioral characteristic quantity information X _i ^paf when the association of the object has been performed. For example, with respect to the time t _i of the detected object i, and the object detected at time t _i-1, and that is the object detected at time t _{i + 1} associated. In this case, the behavior characteristic amount extraction unit 124, as shown in FIG. 5, extracts the action feature amount information based on the partial image of the detected object i at time t _i, the time t _i-1 and time t The behavioral feature amount information of the object detected in _{i + 1 is extracted.} Then, the behavior characteristic amount extraction unit 124, extracted to calculate the average value of these actions feature amount information, by activating reextracted action feature amount information X _i ^paf. Moreover, action feature quantity extracting unit 124, like the method to integrate the action feature amount information X _i ^paf-C and behavioral feature amount information X _i ^paf-F, the maximum element extraction the extracted these actions feature amount information law to calculate the behavior characteristic amount information by applying, by activating the calculated results may be re-extracted action feature amount information X _i ^paf.

In the example shown in FIG. 5, the time _{t i-1,} time _{t i,} and the vector _B t _(i-1) indicating the action feature amount information of the detected object at the time _{t i + 1, B ti,} B t (i + 1) Of the behavior class values included in, the behavior class with a relatively large value is shown in a color different from that of other behavior classes. Vector _{B t (i-1),} B ti, B t (i + 1) results of the integration of the vector indicating the action feature amount information _X ^{i paf} is vector _{B ti} (re-extraction before behavioral characteristic quantity information of the time _{t i} It can be confirmed that it changes from the vector indicating. Note that action feature quantity extraction unit 124, not only the time _{t i,} as the action feature amount information of the detected object at time _{t i-1} and time _{t i + 1,} was re extracted action feature amount information _X ^{i paf} The vector shown may be used.

The behavior feature extraction unit 124 changes the window size of each of the plurality of objects associated by the association unit 125 based on the behavior tendency indicated by the behavior feature corresponding to the object. You may change the number of other objects used to re-extract the behavioral features of the object. For example, the behavior feature amount extraction unit 124 may increase the window size when the behavior tendency shows a static behavior tendency such as stationary, and may decrease the window size when the behavior tendency shows a dynamic behavior such as walking. .. By doing so, the window size can be set to an appropriate window size for the behavioral tendency, so that the accuracy of extracting the behavioral feature amount can be improved.

Further, the behavior feature amount extraction unit 124 is based on the feature amount indicated by the intermediate layer of the neural network, provided that the number of re-extractions is within a predetermined number even when the objects are associated with each other. The behavioral features of the object may be re-extracted. Specifically, the behavioral feature amount extraction unit 124 has a neural network with a partial image corresponding to the detected object i and a partial image corresponding to one or more other objects associated with the object i. enter a acquires the feature quantity indicating the intermediate layer in the neural network, based on the obtained feature amount may be re-extracted action feature amount information X _i ^paf of the object i.

For example, the behavioral feature amount extraction unit 124 inputs the feature amount indicated by the intermediate layer in the neural network when the partial image corresponding to the detected object is input to the neural network, and the partial image corresponding to the related object in the neural network. in the case where input to the average value of the feature quantity indicated by the intermediate layer in the neural network may be re-extracted action feature amount information X _i ^paf of the detected object. By doing so, the object tracking apparatus 1, when the association of the object is not sufficiently, can have ambiguity in the action feature amount information X _i ^paf.

[Object Association]
The association unit 125 is the time when the image is taken based on the position feature amount extracted by the object detection unit 122, the appearance feature amount extracted by the appearance feature amount extraction unit 123, and the behavior feature amount extracted by the behavior feature amount extraction unit 124. The object detected by the object detection unit 122 is associated with each of the plurality of captured images having different values.

The association unit 125 associates the objects by using the detected object i as a node, setting a cost between each node, and solving the minimum cost flow problem. Specifically, first, the association unit 125 defines a _{node y i} indicating the detected object i as shown in the following equation (1).

Subsequently, the association unit 125 calculates the cost for _{each node y i.} The costs include the observation cost _Cobsv (i), the transition cost C _tran (i, j), the start cost _Center (i), and the end cost _Exit (i). J included in the transition cost C _tran (i, j) indicates the index of _{another node y j.} Further, it is assumed that the other node y _j corresponds to the object detected from the captured image captured within a predetermined time from the imaging time of the captured image in which the object i is detected.

Further, the starting cost _Center (i) is given as a constant in advance. The smaller the starting cost _Center (i), the more often new associations are made to _{the node y i.} The end cost _Exit (i) is given as a constant in advance. _{The smaller the exit} cost EXIT (i), the more often no association is made to the _{node y i.}

The association unit 125 _calculates the observation cost Cobsv (i) based on the following equations (2) and (3).

B shown in the equation (2) is a predetermined constant. Cost C _{det (i)} shows the output of the object detector (probability that the object is present at a position indicated by the position characteristic amount information X _i ^loc). α and β are _{parameters for calculating the observation cost Cobsv} (i) that is _{optimal for the cost C det} (i). alpha, beta is a cost C _{det (i)} at position feature quantity information X _i ^loc, set by performing learning based on training data obtained by combining the correct data position of the object indicating whether correct or not NS.

The association unit 125 _{calculates the transition cost C tran} (i, j) based on the following equations (4) and (5).

_C iou represented by formula (4) (i, j), for example, node _y position feature quantity information contained in the _{i X} ⁱ and the bounding box indicated by ^loc, node _{y j} position feature quantity information _{X j} included in The overlap rate with the bounding box indicated by the ^loc. _C app represented by formula (4) (i, j) is the cosine distance between the appearance feature amount information _X ^{j app} in a node _y appearance feature amount information contained in the _{i X} ^{i app} and node _{y j. C} paf represented by formula (4) (i, j) is the cosine distance between action feature quantity information _X ^{j paf} in a node _y action feature quantity information contained in the _{i X} ^{i paf} and node _{y j.}

Function g, _C iou in equation _{(4) (i, j)} , C app (i, j), C paf (i, j) a result of entering the optimal become observed cost _C tran (i, j) It is a non-linear function for calculation. The function g is set by learning based on teacher data that combines a plurality of objects i that are associated in advance and nodes y corresponding to the plurality of objects i detected by the object detection unit 122. Will be done. The function g may be expressed by, for example, a decision tree. In this case, the function g is supposed to learn the parameters in advance by, for example, a boosting algorithm.

The association unit 125 associates the objects by obtaining the result F of the association of the objects shown in the following equation (6).

Here, f is, which indicates whether the _{association, f entr (i), f} obsv (i), f tran (i, j), of _{f exit} (i), either 1 And the others are 0. When f is 0, it indicates that the association is not performed, and when f is 1, it indicates that the association is performed.

^{The association unit 125 obtains the result F *} of the optimum association of each object by optimizing the objective function shown in the following equation (7), that is, by ^{setting F *} to the minimum value. The association unit 125 associates the objects by optimizing the objective function shown in the equation (7) using, for example, a scaling push relabeling algorithm.

The association unit 125 reassociates the objects detected by the object detection unit 122 from each of the plurality of captured images having different captured times based on the behavior feature amount re-extracted by the behavior feature amount extraction unit 124. When the associating unit 125 reassociates the objects, as described above, the associating the objects is performed _{by calculating the cost for each node y i} and optimizing the objective function shown in the equation (7). ..

[Re-extraction of behavioral features and repetition of object association]
The execution control unit 126 tracks the object by alternately and repeatedly executing the re-extraction of the behavior feature amount by the behavior feature amount extraction unit 124 and the re-association of the object by the association unit 125 until a predetermined condition is satisfied. I do. For example, the predetermined condition is that the association of objects by the association unit 125 has converged.

Specifically, the execution control unit 126 associates the objects with the association unit 125, and as a result, the objects associated with the number of objects associated with the association after the association is performed before the association is performed. The re-extraction of the behavioral feature amount by the behavioral feature amount extraction unit 124 and the association of the objects by the association unit 125 are alternately and repeatedly executed until the ratio of the number of the objects becomes a predetermined ratio or more (for example, 90% or more). That is, the execution control unit 126 re-extracts the behavior feature amount by the behavior feature amount extraction unit 124 until the change (increment) in the number of associated objects before and after the association of the objects by the association unit 125 becomes small. , The object association by the association unit 125 is repeatedly executed alternately. By doing so, the object tracking device 1 can associate the objects until the association of the objects converges.

It should be noted that the number of times that the behavioral feature amount extraction unit 124 re-extracts the behavioral feature amount and the association unit 125 alternately repeatedly executes the object association is determined in advance, and the execution control unit 126 determines the number of times based on the number of times. The re-extraction of the behavioral feature amount by the behavioral feature amount extraction unit 124 and the association of the objects by the association unit 125 may be alternately and repeatedly executed.

[Process flow in object tracking device 1]
Subsequently, the processing flow in the object tracking device 1 will be described. FIG. 6 is a flowchart showing a processing flow in the object tracking device 1 according to the present embodiment.

First, the acquisition unit 121 acquires a plurality of captured images captured by the imaging device 2 that images a predetermined area at each of the plurality of times (S1).
Subsequently, the object detection unit 122 detects the object reflected in each of the plurality of captured images by extracting the position feature amount corresponding to the object from each of the plurality of captured images acquired by the acquisition unit 121 (S2). ..

Subsequently, the visible feature amount extraction unit 123 extracts the visible feature amount of the object detected by the object detection unit 122 (S3).
Subsequently, the behavior feature amount extraction unit 124 extracts the behavioral feature amount of the object detected by the object detection unit 122 (S4).

Subsequently, the association unit 125 is an object based on the position feature amount extracted by the object detection unit 122, the appearance feature amount extracted by the appearance feature amount extraction unit 123, and the behavior feature amount extracted by the behavior feature amount extraction unit 124. The detection unit 122 associates the detected objects (S5).

Subsequently, the behavior feature extraction unit 124 uses the behavior feature amount extraction unit 124 based on the object and one or more other objects associated with the object for each of the plurality of objects associated with the association unit 125. Is re-extracted (S6).

Subsequently, the association unit 125 is based on the position feature amount extracted by the object detection unit 122, the appearance feature amount extracted by the appearance feature amount extraction unit 123, and the behavior feature amount re-extracted by the behavior feature amount extraction unit 124. The object detection unit 122 associates the detected object (S7).

Subsequently, the execution control unit 126 determines whether or not the object association by the association unit 125 has converged (S8). When the execution control unit 126 determines that the object associations have converged, it ends the process according to this flowchart, and when it determines that the object associations have not converged, it shifts the process to S6.

[Effect in this embodiment]
As described above, the object tracking device 1 according to the present embodiment detects an object from a plurality of captured images obtained by capturing a predetermined area at each of a plurality of times, and converts the object into a partial image of the captured image corresponding to the detected object. Based on this, the behavioral feature amount of the object is extracted, and the objects detected from each of the plurality of captured images having different imaging times are associated with each other based on the extracted behavioral feature amount. Then, the object tracking device 1 has a behavioral feature amount based on the partial image corresponding to the object and the partial image corresponding to the other object associated with the object for each of the plurality of associated objects. Is re-extracted, and based on the re-extracted behavioral feature amount, the objects detected from each of the plurality of captured images having different imaging times are associated again.

By doing so, the object tracking device 1 can increase the accuracy of the behavioral feature amount of the object corresponding to each of the associated objects, so that the object can be tracked with high accuracy.

Further, the object tracking device 1 tracks the object by alternately and repeatedly executing the re-extraction of the behavioral feature amount and the re-association of the object until a predetermined condition is satisfied. As a result, the object tracking device 1 can further improve the accuracy of the behavioral features of each object based on the behavioral features indicated by the associated objects, so that the tracking accuracy of the objects can be further improved. Can be done.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist thereof. be. For example, in the above-described embodiment, the association unit 125 uses the position feature amount extracted by the object detection unit 122, the appearance feature amount extracted by the appearance feature amount extraction unit 123, and the behavior feature amount extracted by the behavior feature amount extraction unit 124. It was decided to associate objects based on, but it is not limited to this.

The association unit 125 may associate the objects detected by the object detection unit 122 based only on the behavior feature amount extracted by the behavior feature amount extraction unit 124. Further, the association unit 125 is based on either the position feature amount extracted by the object detection unit 122 or the visible feature amount extracted by the visible feature amount extraction unit 123, and the behavioral feature amount extracted by the behavioral feature amount extraction unit 124. The objects detected by the object detection unit 122 may be associated with each other.

Further, the acquisition unit 121 has acquired a plurality of captured images of 4 or more captured by the imaging device 2 at a plurality of times of 4 or more, but the present invention is not limited to this. The acquisition unit 121 may acquire three captured images captured by the imaging device 2 at each of the three times. In this case, the behavioral feature amount extraction unit 124 re-extracts the behavioral feature amount and the association unit 125 re-extracts the behavioral feature amount of the object only once, respectively, and the behavioral feature amount is re-extracted. The re-extraction of the behavioral features and the association of the objects for which the behavioral features have been re-extracted may not be executed multiple times.

Further, in the above embodiment, the object is described as an example of a person such as a clerk or a customer who acts in the store, but the present invention is not limited to this. For example, when the captured image is a captured image that captures the entire town, not only a person but also a moving object such as a vehicle may be an object. In this case, the object tracking device 1 may use various feature quantities according to the type of the object.

In addition, all or part of the device can be functionally or physically distributed / integrated in any unit. Also included in the embodiments of the present invention are new embodiments resulting from any combination of the plurality of embodiments. The effect of the new embodiment produced by the combination also has the effect of the original embodiment.

1 ... Object tracking device, 11 ... Storage unit, 12 ... Control unit, 121 ... Acquisition unit, 122 ... Object detection unit, 123 ... Visible feature amount extraction unit, 124 ... -Behavior feature extraction unit, 125 ... association unit, 126 ... execution control unit, 2 ... imaging device

Claims (13)

An acquisition unit that acquires a plurality of three or more captured images captured by an imaging device that captures a predetermined area at a plurality of three or more times, respectively.
An object detection unit that detects an object reflected in the captured image from each of the plurality of captured images acquired by the acquisition unit, and an object detection unit.
A behavioral feature amount extraction unit that extracts a behavioral feature amount indicating the behavioral feature amount of the object based on a partial image of the captured image corresponding to the object detected by the object detection unit, and a behavioral feature amount extraction unit.
An association unit that associates the object detected by the object detection unit from each of the plurality of captured images having different captured times based on the behavior feature amount extracted by the behavior feature amount extraction unit.
With
The behavior feature extraction unit corresponds to each of the partial image corresponding to the object and one or more other objects associated with the object for each of the plurality of objects associated with the association unit. The behavioral feature amount is re-extracted based on the partial image to be performed.
The association unit associates the object detected by the object detection unit with each of the plurality of captured images having different captured times based on the behavior feature amount re-extracted by the behavior feature amount extraction unit. Do it again
Object tracking device. The acquisition unit acquires a plurality of four or more captured images captured by the imaging device at each of the four or more time periods.
Execution control for tracking an object by alternately and repeatedly executing the re-extraction of the behavior feature amount by the behavior feature amount extraction unit and the re-association of the object by the association unit until a predetermined condition is satisfied. With more parts,
The object tracking device according to claim 1. As a result of associating the objects by the association unit, the execution control unit has a ratio of the number of objects associated before the association to the number of objects associated after the association. The re-extraction of the behavioral feature amount by the behavioral feature amount extraction unit and the association of the objects by the association unit are alternately and repeatedly executed until the ratio becomes a predetermined ratio or more.
The object tracking device according to claim 2. The behavior feature extraction unit determines the behavior feature of the object based on the behavior tendency indicated by the behavior feature corresponding to the object for each of the plurality of objects associated with the association. Change the number of other objects used for re-extraction,
The object tracking device according to claim 2 or 3. The object detection unit detects the object by specifying an object position indicating the position of the object reflected in the captured image.
The behavior feature amount extraction unit extracts the behavior feature amount based on the partial image corresponding to the object position specified by the object detection unit.
The object tracking device according to any one of claims 1 to 4. The association unit further associates the object detected by the object detection unit from each of the plurality of captured images having different captured times, based on the object position specified by the object detection unit.
The object tracking device according to claim 5. The object detection unit further includes a visual feature amount extraction unit that extracts a visual feature amount that indicates the appearance of the object in the captured image.
Based on the visible feature amount extracted by the visible feature amount extraction unit, the associating unit further associates the object detected by the object detection unit from each of the captured images having different captured times.
The object tracking device according to any one of claims 1 to 6. The behavioral feature amount extraction unit captures the partial image of the captured image corresponding to the object detected by the object detection unit and an image captured at a time before or after the time at which the captured image is captured. The behavioral feature amount is extracted based on the difference between the image and the partial image.
The object tracking device according to any one of claims 1 to 7. The behavioral feature amount extraction unit is a first partial image that is a partial image including the object detected by the object detection unit, and the partial image that includes the object and has a larger display area than the first partial image. The behavioral feature amount is extracted based on the second partial image.
The object tracking device according to any one of claims 1 to 8. The behavior feature extraction unit is one or more of the above, in which the partial image corresponding to the object is associated with the object in a neural network that outputs the behavior feature in response to the input of the partial image. The partial image corresponding to the other object is input, and the behavioral feature of the object is re-extracted based on the plurality of behavioral features output from the neural network.
The object tracking device according to any one of claims 1 to 9. The behavior feature extraction unit inputs the partial image corresponding to the object into the neural network that outputs the behavior feature in response to the input of the partial image, and obtains the feature indicated by the intermediate layer in the neural network. Acquire and extract the behavioral feature of the object based on the acquired feature.
The object tracking device according to any one of claims 1 to 10. The behavior feature extraction unit is one or more of the above, in which the partial image corresponding to the object is associated with the object in a neural network that outputs the behavior feature in response to the input of the partial image. The partial image corresponding to the other object is input, the feature amount indicated by the intermediate layer in the neural network is acquired, and the behavioral feature amount of the object is re-extracted based on the acquired feature amount.
The object tracking device according to any one of claims 1 to 10. Computer runs,
A step of acquiring three or more captured images captured by an imaging device that captures a predetermined area at a plurality of three or more times, respectively.
A step of detecting an object reflected in the captured image from each of the acquired plurality of captured images, and
A step of extracting a behavioral feature amount indicating the behavioral feature amount of the object based on a partial image of the captured image corresponding to the detected object, and a step of extracting the behavioral feature amount.
Based on the extracted behavioral features, a step of associating the objects detected from each of the plurality of captured images having different captured times, and a step of associating the objects.
For each of the plurality of associated objects, the behavioral feature amount is based on the partial image corresponding to the object and the partial image corresponding to each of one or more other objects associated with the object. And the steps to re-extract
A step of reassociating the objects detected from each of the plurality of captured images having different captured times based on the re-extracted behavioral features.
Object tracking method with.

Images