JP7198661B2 - Object tracking device and its program - Google Patents

Description

The present invention relates to an object tracking device and program for tracking an object using an infrared image and a visible image.

In recent years, with the progress of video analysis technology, various applications using cameras have been proposed. The development of this technology is particularly remarkable in video analysis of sports scenes. For example, the Hawkeye system for tennis, which is also used at Wimbledon, tracks a tennis ball three-dimensionally using images from a plurality of fixed cameras, and performs IN/OUT determination. Also, at the 2014 FIFA World Cup, goal line technology was used to analyze images from several fixed cameras and automate goal determination. Furthermore, a TRACAB system is also known, in which a large number of stereo cameras are installed in a soccer stadium and all players on the field are tracked in real time.

These video analysis techniques are often based on the premise of using video captured by a camera with a temporal resolution of 30 frames per second (fps). For example, if an object moving at such a high speed that it is difficult to see with the naked eye, such as the point of a fencing sword or a badminton shuttlecock, is photographed, extreme motion blur occurs in the image of the object (marked α in FIG. 16). Therefore, it is extremely difficult to accurately measure the object position only from the image. In this case, motion blur can be reduced by using a high-speed camera exceeding 30 fps or by increasing the shutter speed. On the other hand, the high-speed camera is expensive, and there is a problem that the brightness of the image decreases when the shutter speed is increased.

Under such constraints, a conventional technique has been proposed that uses an infrared camera to robustly track a high-speed moving object (Patent Document 1). This conventional technology measures the position of a tracked target by attaching a retroreflective tape to the tracked target, irradiating infrared light from an infrared camera, and detecting the reflected light on an infrared image. is. In this prior art, detection is performed on an infrared image, thereby reducing noise that causes erroneous detection in a visible image and tracking an object with high accuracy.

JP 2018-78431 A

In the above-described prior art, when the object to be tracked is moving at high speed, or when the reflective tape does not face the infrared camera, the reflected light from the reflective tape becomes weak, and the object to be tracked is displayed on the infrared image. Difficult to detect. Henceforth, failure to detect a tracked object may be referred to as "lost". Further, in the conventional technology, when a plurality of tracked objects are being tracked and the tracked objects come close to each other, or when all the tracked objects are lost and redetected, the tracked objects may be replaced. In this case, with the conventional technology, it is extremely difficult to draw an accurate trajectory, and the trajectory may be replaced.

Accordingly, it is an object of the present invention to provide an object tracking device and its program that can suppress the replacement of trajectories.

In view of the above-described problems, an object tracking device according to the present invention provides an infrared image obtained by photographing an infrared light marker attached to each moving object with infrared light, and a person moving each object. An object tracking device for tracking an object using a visible image captured with visible light, comprising infrared light detection means, joint position detection means, feature vector calculation means, attribute information generation means, and a trajectory and generating means.

In such an object tracking device, the infrared light detection means detects the position of the infrared light marker from the infrared image as the position of the object.
The joint position detection means detects each joint position of the person from the visible image.
The feature vector calculation means calculates feature vectors from the position of the object to each joint position. This feature vector represents the relationship between the position of the object to be tracked and the pose of the person.

The attribute information generating means selects a person corresponding to the object according to the feature vector using a classifier that has previously learned the relationship between the position of the object and the positions of each joint, and generates attribute information indicating the correspondence between the object and the person. Generate.
The trajectory generating means generates the trajectory of the object based on the position and attribute information of the object.
In this way, when tracking an object, the object tracking device uses the attribute information indicating the correspondence between the object and the person, so it is possible to suppress the change of the trajectory.

The present invention can also be implemented by a program that causes hardware resources such as a CPU, memory, and hard disk provided in a computer to operate cooperatively as the object tracking device described above.

According to the present invention, when tracking an object, the attribute information indicating the correspondence between the object and the person is used, so that the trajectory can be suppressed from being replaced. Thus, according to the present invention, an accurate object trajectory can be generated and tracking robustness can be improved.

1 is a schematic configuration diagram of an object tracking system according to an embodiment; FIG. It is explanatory drawing of the sword point in embodiment. In an embodiment, it is a figure showing an example of an infrared picture. 1 is a block diagram showing the configuration of an object tracking device according to an embodiment; FIG. FIG. 4 is an explanatory diagram of joint points of a person in an embodiment; FIG. 4 is a diagram showing an example of a visible image in the embodiment; FIG. 10 is an explanatory diagram illustrating detection of joint points in the embodiment; FIG. 10 is an explanatory diagram illustrating selection of a player's joint points in the embodiment; FIG. 4 is an explanatory diagram illustrating calculation of a feature vector in the embodiment; FIG. 4 is an explanatory diagram for explaining learning of a discriminator in an embodiment; FIG. 4 is an explanatory diagram for explaining determination by a discriminator in an embodiment; FIG. 10 is an explanatory diagram illustrating drawing of a trajectory in the embodiment; 4 is a flow chart showing the operation of the object tracking device according to the embodiment; It is an example of the image which visualized likelihood distribution in an Example. 3 is another example of an image in which the likelihood distribution is visualized in the example. 1 is an explanatory diagram illustrating motion blur in a fencing video in the prior art; FIG.

(embodiment)
[Overview of object tracking system]
BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.
An overview of an object tracking system 1 according to an embodiment of the present invention will be described with reference to FIG.
In the following embodiments, in fencing, the point of a sword (object) held by a player (person) will be described as a tracking target. During fencing, the tips of both athletes' swords are often moving at high speed.

The object tracking system 1 uses a visible/infrared coaxial camera 20 capable of photographing visible light and infrared light on the same optical axis, combines a visible light image V and an infrared image I, and moves 2 at high speed. It traces the position of the tip of the book and draws its trajectory T (T ¹ , T ² ). As shown in FIG. 1 , the object tracking system 1 includes an infrared light projector 10 , a visible/infrared coaxial light camera 20 , and an object tracking device 30 .

The infrared light projector 10 is a general light projector that projects infrared light.
As shown in FIG. 2, the infrared light projected by the infrared light projector 10 is reflected by a reflective tape (infrared light marker) 91 attached to the tips of the swords 90 of both athletes, resulting in a visible/infrared coaxial light, which will be described later. Photographed by the optical camera 20 .

The reflective tape 91 reflects infrared rays from the infrared light projector 10 . One or more reflective tapes 91 may be attached to the tip 90, and the size and number of the tapes are not particularly limited. In the example of FIG. 2, the sword tip 90 has a piece of rectangular reflective tape 91 attached to its side surface. Here, the sword tip 90 may be provided with a piece of reflective tape 91 added to the opposite side, or a band-shaped reflective tape 91 may be wound around the side (not shown).

The visible/infrared coaxial camera 20 captures visible light and infrared light on the same optical axis to generate a visible image V and an infrared image I having the same number of pixels. In this embodiment, the visible/infrared coaxial camera 20 generates a visible image V ( FIG. 5 ) of fencing competition and an infrared image I of the reflective tape 91 of the tip 90 . As shown in FIG. 3, in the infrared image I, only the two reflective tapes 91 are photographed, while other athletes and the like are not photographed (indicated by broken lines). Further, since the image coordinates of the tip of the sword 90 of the visible image V and the reflective tape 91 of the infrared image I correspond to each other, the trajectory T can be drawn without converting the viewpoint in the three-dimensional space.

The object tracking device 30 uses the infrared image I and the visible image V input from the visible/infrared coaxial camera 20 to track the tips 90 of both players. Then, the object tracking device 30 draws the tracked trajectories T ¹ and T ² of the tip of the sword 90 of both athletes in different colors, and synthesizes the drawn trajectories T ¹ and T ² with the visible image V to obtain a trajectory composite image. Generate F.
In ^FIG . 1, the trajectory T1 of the tip 90 held by the player on the left is indicated by a broken line, and the trajectory T2 of the tip 90 held by the player on the right is indicated by ^a dashed line.

[Configuration of object tracking device]
The configuration of the object tracking device 30 will be described with reference to FIG.
As shown in FIG. 4 , the object tracking device 30 includes infrared light detection means 31 , human posture acquisition means 33 , object identification means 35 and object tracking means 37 .

In this embodiment, the object tracking device 30 receives infrared images I and visible images V of frames 1, . . . , t−1, t, . Suppose that an image V is subjected to sequential processing. Hereinafter, the current frame (current frame) is defined as _t , the infrared image I of the current frame _t is defined as the infrared image It, and the visible image V of the current frame is defined as the visible image Vt.

The infrared light detection means 31 detects the position of the tip 90 (reflective tape ₉₁ ) from the infrared image It. An example of a technique for detecting the position of the tip of the sword by the infrared light detection means 31 will be described below.

<Detection method of sword tip position>
First, the infrared light detection means 31 uses the following equation (1) to obtain a binary infrared image of the infrared image I t of the current frame and the infrared image I _{t -1} of the previous frame. Only the region _Mt of the moving object is extracted by generating the difference image. That is, the infrared light detection means 31 detects the luminance value I ^xy _t of the pixel (x, y) of the infrared image I _t and the luminance value I ^xy _t of the pixel (x, y) of the infrared image I _t−1 . A moving object region M ^xy _t whose difference from ₋₁ exceeds a preset threshold value R_bri is extracted as a candidate blob.

where x,y represent the horizontal and vertical image coordinates. Also, the threshold value R_bri is preset with an arbitrary value. Also, '0' in Equation (1) represents the minimum luminance value, and '255' represents the maximum luminance value.
Note that the infrared light detection means 31 generates the binary infrared difference image M ^xy _t in order to suppress the generation of stationary noise blobs, but the area with high luminance in the infrared image I _t is extracted as a candidate blob. You may

Next, the infrared light detection means 31 applies morphology processing to the extracted candidate blobs to eliminate noise blobs in small areas. This morphology processing is processing for eliminating noise blobs in a small area by performing inter-image operations between an image group obtained by shifting an image in several directions pixel by pixel and the original image.

Next, the infrared light detection means 31 performs labeling processing on the candidate blobs remaining after the morphology processing. This labeling process is a process of assigning labels (numbers) to candidate blobs.
Next, the infrared light detection means 31 obtains the position, area, and shape feature amount of the candidate blob subjected to the labeling process. Here, the position of the candidate blob is the center position or barycentric position of the candidate blob. Also, the shape feature amount of the candidate blob is the degree of circularity or the aspect ratio of the circumscribing rectangle.

Next, the infrared light detection means 31 eliminates candidate blobs that are not within the range from the preset minimum area to the maximum area. Then, the infrared light detection means 31 deletes candidate blobs whose shape feature amount is not within the preset range. Furthermore, when the number of candidate blobs exceeds the upper limit number of objects, the infrared light detection unit 311 leaves the positions of two candidate blobs with large areas as the positions S ¹ and S ² of the tip 90, and leaves other candidates. Erase the blob. S ^m (S ¹ , S ² ) represents the position of the sword tip 90 to which the left and right attribute information described later is not added (mε1, 2).
After that, the infrared light detection means 31 sends the positions of the ^two reflective tapes ₉₁ detected from the infrared image It to the object identification means 35 ⁽ feature vector calculation means 351) as positions S1 and S2 of the tip 90. Output.

The human pose acquisition means 33 acquires the pose of a person from the visible image _Vt , and includes a person pose detection means (joint position detection means) 331 and a person selection means 333 .

The human pose detection means 331 detects each joint point (joint position) of the person from the visible image _Vt as the pose of the person. Here, the human pose detection means 331 can detect the joint points of the person by any method, and may automatically detect the joint points from the visible image _Vt , or manually detect the joint points on the visible image _Vt . may be specified.

In the present embodiment, the human posture detection unit 331 uses "OpenPose", which is one of the general posture measurement methods (reference document 1).
Reference 1: ZheCao, Tomas Simon, Shih-EnWei, YaserSheikh, ”Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition 2017 (CVPR2017), pp.7291- 7299

This posture measurement method is a method of measuring a person's posture using deep learning, and detects 18 joint points of each person from the visible image _Vt . Hereinafter, as shown in FIG. 5, the joint points of each person are represented by B ⁿ _i . A superscript n represents the identification number of a person included in the visible image Vt _{( nεN} ), and let N be the total number of people included in the visible image Vt. The subscript i represents the identification number of the joint point (i=0 to 17).
Note that FIG. 5 omits the identification number n and shows dashed lines connecting adjacent joint points B ⁿ _i .

The joint points B _i include upper body joint points B ₀ to B _{7 and} B ₁₄ to B ₁₇ and lower body hip joints B ₈ to B ₁₃ including hip joints B ₈ and B ₁₁ . Although non-joint points such as the joint points B ₁₄ to B ₁₇ of the head are included, they are also treated as joint points because they have image features such as the eyes and nose.

For example, assume that the visible image _Vt in FIG. This visible image _Vt is a video of a fencing match, and includes eight persons H, including two players ^HL and ^HR , one referee ^HJ , and five spectators ^HG . (N=8). Here, in the posture measurement method of Reference 1, since the person H whose joint points B ⁿ _i are to be detected cannot be specified in the visible image V _t , the joint points B ⁿ _i of all persons H are detected. That is, the human posture detection means 331 detects the joint points B ⁿ _i of all the people H included in the visible image V _t as shown in FIG. 7 . In addition, in FIG. 7, only some of the joint points B ⁿ _i are illustrated with reference numerals for easy viewing of the drawing.
After that, the person posture detection means 331 outputs the visible image _Vt and the joint points ^Bn _i of all the persons H detected from this visible image _Vt to the person selection means 333 .

As described above, the joint points _Bni of all persons H included in the _visible image ^Vt are detected. Also, in a fencing game video, the players ^HL and ^HR are often shot in a larger size than the spectators ^HG and the referee ^HJ . Therefore, the person selection means 333 calculates the torso length ^ln based on the joint points ^Bn _i of all the persons H, as shown in the following equation (2), and presets the calculated torso length ^ln in descending order. The number of people H selected is selected.

When the joint points ^Bn _i are detected by the _posture measurement method of Reference ₁ , as shown _in FIG. be. Therefore, as shown in equation (2), the average value of the length of the vector from neck joint B ₁ to one hip joint B ₈ and the length of the vector from neck joint B ₁ to the other hip joint B ₁₁ is the torso The length ^ln .

In addition, since the fencing match is played by two players ^HL and ^HR , two players ^HL and ^HR having a long torso length ^ln should be selected as shown in FIG. The neck joint B ₁ , hip joints B ₈ and B ₁₁ , and the number of people to be selected are preset in the person selection means 333 .

In addition, in a _fencing match, the positions of the ^left and right ^{players HL} and ^{HR are} not ^interchanged . can be described. In this embodiment, the identification number n indicating the player HL on the left side is replaced with ^L , and the identification number n indicating the player HR on the right side is replaced with ^R.
After that, the person selection means 333 selects the joint point B ^Li and the torso length l ^L of the left player H ^L and the joint point B ^R _i and the torso length _l ^R of the right player H ^R as the object identification means 35 ( Output to the feature vector calculation means 351).

The object identification means 35 identifies whether each of the two sword tips 90 detected from the fencing game video corresponds to the left player ^HL or the right player ^HR . Means 351 and attribute information generating means 353 are provided.

The feature vector calculation means 351 calculates feature vectors from the positions S ¹ and S ² of the tip of the sword 90 to the joint points B ^Li and B ^R _i of the _players ^HL and ^HR . As shown in FIG. 9, the feature vector calculation means 351 calculates a feature vector from the position _S1 of the ^first sword tip 90 to each joint point ^BRi of the player ^HR on the right side. In addition, in FIG. 9, the feature vector is illustrated with the arrow of the chain double-dashed line. The feature vector calculation means 351 also calculates a feature vector from the position _S1 of the tip of the sword 90 to ^each joint point ^BLi of the player ^HL on the left side. In this way, the feature vector is directed from the position S1 of the tip of the sword ⁹⁰ to both the left and right players ^HL and ^HR , so it becomes a robust feature quantity that takes into consideration the relative positions of the left and right players ^HL and ^HR .

Although not shown, the feature vector calculation means 351 calculates each joint point B ^R of the player H ^R on the right side from the position S ² of the second sword tip 90 in the same way as the position S ¹ of the first sword tip 90 . A ^feature _vector up to _i and a feature vector from the position S2 of the tip of the sword 90 to ^each joint point ^BLi of the player HL on the left side are calculated.

This feature vector is represented by the following equation (3), and since there are 18 joint points B ^L _i and B ^R _i each, it becomes a 36-dimensional feature amount. Also, depending on the zoom amount of the visible/infrared coaxial camera 20, the sizes of the players ^HL and ^HR change within the visible image _Vt . Therefore, the feature vector calculation means 351 normalizes (divides) the players ^HL and ^HR by the torso lengths ^lL and ^lR of the players HL and ^HR , as shown in Equation (3), to obtain the sizes of the players ^HL and HR. Invariant feature vectors can be calculated.

After that, the feature vector calculating means 351 outputs the calculated feature vector and the positions S ¹ and S ² of the tip 90 to the attribute information generating means 353 .

The attribute information generating means 353 selects the players ^HL and ^HR corresponding to the positions S ¹ and S ² of the tip 90 by using a classifier learned in advance, and determines the correspondence between the tip 90 and the players ^HL and ^HR . It generates attribute information that indicates the relationship. That is, the attribute information is information that associates the two players ^HL and ^HR with the positions S ¹ and S ² of the sword points 90 of the swords that the players ^HL and ^HR are moving.

This attribute information generation means 353 operates in two operation modes. A first operation mode is a learning mode in which the attribute information generating means 353 learns a discriminator. The second operation mode is a selection mode in which the attribute information generating means 353 selects the players ^HL and ^HR corresponding to the positions S ¹ and S ² of the tip 90 using the learned discriminators. In this embodiment, the user of the object tracking device 30 manually switches between the two operation modes.

<Learning mode>
First, the learning mode of the attribute information generating means 353 will be described.
The discriminator has learned the feature vector of FIG. 9, that is, the relationship between the positions S ¹ and S ² of the tip of the sword 90 and the joint points B ^L _i and B ^R _i of the players H ^L and H ^R . In this embodiment, the attribute information generating means 353 learns a regression model discriminator using a support vector machine (SVM).

At this time, as shown in FIG. 10, the attribute information generating means 353 learns classifiers for each of the left and right players ^HL and ^HR . In FIG. 10, SVM regression ( ^L ) is the discriminator corresponding to the player HL on the left, and SVM regression ( ^R ) is the discriminator corresponding to the player HR on the right. The learning data for SVM regression ( ^L ) is obtained by manually setting the positions S ^L and S ^R of the tip 90 on the visible image V, and ^obtaining a score of 1.0 ( A positive example), and a score of -1.0 (negative example) at the position S ^R of the tip 90 corresponding to the player H ^R on the right side. The positions S ^L and S ^R of the tip 90 represent the positions of the tip 90 to which the left and right attribute information is added. Similar to the SVM regression ( ^L ), the learning data for the SVM regression (R) is a score of -1.0 (negative example) at the position SL of the tip 90 corresponding to the player ^HL on the left side, and the score for the player ^HR on the right side. A score of 1.0 (positive example) may be given at the corresponding position ^SR of the tip 90 . In general, if 100 sets or more of training data are prepared, a highly accurate discriminator can be learned.

<Selection mode>
Next, the selection mode of the attribute information generating means 353 will be described.
The discriminator of the regression model outputs the determination result as a numerical value (likelihood) instead of binary determination such as success/failure, positive/negative, or true/false. That is, the regression model discriminator outputs the likelihood that each of the two sword tips 90 belongs to the left player ^HL and the likelihood that it belongs to the right player ^HR . Therefore, as shown in FIG. 11, the attribute information generating means 353 inputs the feature vector to the SVM regression ( ^L ) corresponding to the player HL on the left side, and the likelihood indicating that it is the player ^HL on the left side, Then, the likelihood of being the player ^HR on the right side is calculated. Furthermore, the attribute information generation means 353 inputs the feature vector to the SVM regression ( ^R ) corresponding to the right player HR, and the likelihood indicating that it is the left player ^HL and the right player ^HR . Calculate the likelihood that indicates that In this way, the attribute information generating means 353 calculates a total of four likelihoods corresponding to the left and right players ^HL and ^HR using classifiers corresponding to the left and right players ^HL and ^HR .

Next, the attribute information generating means 353 selects the one with the highest likelihood among the four likelihoods. That is, the attribute information generating means 353 selects the combination with the highest likelihood among the four combinations of the two sword tips 90 and the left and right players ^HL and ^HR . Therefore, the attribute information generating means 353 can inevitably select combinations of the remaining sword tips 90 and the remaining players ^HL and ^HR .

Advantages of calculating four types of likelihoods will be described below in comparison with binary determination.
When binary judgment (classification model) is applied to classifiers corresponding to left and right players ^HL and ^HR , a contradictory judgment result is produced that the same sword tip 90 corresponds to both players ^HL and ^HR . There is For example, for the same sword tip 90, the SVM regression ( ^L ) may be determined to be that of the left player HL and the SVM regression ( ^R ) may be determined to be that of the right player HR. The truth is unknown. On the other hand, since the attribute information generating means 353 calculates a numerical value of likelihood instead of binary determination, it is possible to select the combination of the tip 90 and the players ^HL and ^HR that gives the highest likelihood. There is no contradictory determination result such as

Next, the attribute information generating means 353 generates attribute information indicating the correspondence relationship between each of the two sword tips 90 and the left and right players H ^L and ^HR , and converts the generated attribute information to the position S ¹ of the sword tip 90 . , ^S2 . Then, the attribute information generation means 353 outputs the positions S ^L and S ^R of the tip 90 to which the attribute information is added to the object tracking means 37 (trajectory drawing means 371).

Returning to FIG. 4, the description of the configuration of the object tracking device 30 is continued.
The object tracking means 37 tracks an object, and includes a visible image storage means 371 and a trajectory drawing means (trajectory generating means) 373 .

The visible image storage means 371 is storage means such as a memory, HDD (Hard Disk Drive), or SSD (Solid State Drive) for storing the visible image _Vt input from the visible/infrared coaxial camera 20 . The visible image _Vt accumulated by the visible image accumulation means 371 is referred to by the trajectory drawing means 373, which will be described later.

The trajectory drawing means 373 draws the trajectory of the tip 90 on the visible image _Vt accumulated in the visible image accumulating means 371 while referring to the attribute information. At this time, the trajectory drawing means 373 refers to the attribute information added to the positions ^{S L} and SR of the tip 90, so that the replacement of the trajectory T can be suppressed and the correct trajectory can be drawn.

For example, the trajectory drawing means 373 presets different colors for the tips 90 of the left and right players ^HL and ^HR , such as red for the tip of the player ^HL on the left and green for the tip of the player ^HR on the right. Then, the trajectory drawing means 373 generates a trajectory composite image _Ft by synthesizing the visible image Vt and the trajectory _T , as shown in FIG. In this trajectory composite image _Ft , the trajectory T1 of the tip 90 held by the player ^HL on the left and the trajectory T2 of the tip ⁹⁰ held by the player ^HR on the right ^are synthesized by CG.
After that, the trajectory drawing means 373 outputs the trajectory composite image _Ft to an external device (for example, display).

[Operation of object tracking device]
The operation of the object tracking device 30 will be described with reference to FIG.
As shown in ^FIG . ¹³ , in step S1, the infrared light detection means 31 detects the positions S1 and S2 of the tip ₉₀ from the infrared image It.
For example, the infrared light detection means 31 generates a binary infrared difference image and applies morphological processing to the extracted candidate blobs. Next, the infrared light detection unit 311 performs labeling processing on the candidate blobs remaining after the morphology processing, and obtains the position, area, and shape feature amount of the candidate blobs. Then, the infrared light detection means 31 performs filtering based on the area and shape feature amount, and detects the positions of two candidate blobs having a large area as the positions S ¹ and S ² of the tip 90 .

In step S2, the human posture detection means 331 detects the joint points ^Bn _i of all the people H included in the visible image _Vt . For example, each joint point B ⁿ _i of the person H is detected using “OpenPose” which is one of the general posture measurement methods.
In step S3, the person selection means 333 selects two players ^HL and ^HR having the longest torso length ^ln among all persons H detected in step S2.

In step S4, the feature vector calculation means 351 calculates from the positions S ¹ and S ² of the tip of the sword 90 detected in step S1 to the joint points B ^L _i and B ^R _i of the players ^HL and ^HR selected in step S3. Calculate the feature vector of . At this time, the feature vector calculating means 351 normalizes the torso lengths ^ln of the players ^HL and ^HR .

^In step S5, the attribute information generating means ^{353 selects the players HL and HR} corresponding to the positions S1 and S2 of the tip 90 by using a pre-learned discriminator. Attribute information indicating the correspondence with ^R is generated.
For example, the attribute information generating means 353 uses classifiers of two regression models corresponding to the left and right players ^HL and ^HR to calculate four likelihoods corresponding to the left and right players ^HL and ^HR . Then, the attribute information generating means 353 selects the combination with the highest likelihood among the four combinations of the two sword tips 90 and the left and right players ^HL and ^HR . Further, the attribute information generating means 353 generates attribute information indicating the correspondence between the two sword tips 90 and the left and right players ^HL and ^HR , and sends the generated attribute information to positions S ¹ and S ² of the sword tips 90 . Append to

In step S6, the trajectory drawing means 373 draws the trajectory _T of the tip 90 on the visible image Vt while referring to the attribute information generated in step S5. For example, the trajectory drawing means 373 generates a trajectory composite image _Ft by synthesizing the visible image Vt and the trajectory _T , as shown in FIG.

[Action/effect]
As described above, when tracking the tip 90, the object tracking device 30 uses the attribute information indicating the correspondence between the tip 90 and the players ^HL and ^HR , so that the trajectories can be prevented from being changed. Thus, the object tracking device 30 can generate an accurate trajectory of the tip 90 and improve tracking robustness.

(Modification)
Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design changes and the like without departing from the gist of the present invention.

In the above-described embodiment, the use of a visible/near-infrared coaxial camera has been described, but the present invention is not limited to this. For example, in the present invention, a visible/near-infrared simultaneous imaging camera and a visible/near-infrared multi-wavelength camera can be used.

[Visible and near-infrared simultaneous shooting camera]
The visible/near-infrared simultaneous shooting camera uses a four-wavelength spectral prism that separates IR (near-infrared light) in addition to RGB, and shoots with a sensor for each wavelength, a total of four sensors. This visible/near-infrared simultaneous photographing camera can separately output a visible image from an RGB sensor and an infrared image from an IR sensor. That is, the object tracking device can acquire a visible image and an infrared image from the visible/near-infrared simultaneous photographing camera and output a trajectory composite image, as in the above-described embodiment.

[Visible/near-infrared multi-wavelength camera]
The visible/near-infrared multi-wavelength camera is a camera that uses a multi-wavelength spectroscopic prism that separates three wavelengths in the near-infrared region in addition to the three colors of RGB. Usually, a camera is equipped with an IR cut filter or a visible light cut filter to extract wavelengths indicated by visible spectral characteristics or near-infrared spectral characteristics, and obtains an image of only visible light or near-infrared light. However, the visible/near-infrared multi-wavelength camera is not equipped with an IR cut filter or a visible light cut filter, and extracts all wavelengths indicated by the basic spectral characteristics. Generate an infrared composite image.

Therefore, when using a visible/near-infrared multi-wavelength camera, the object tracking system may include a visible/infrared separation device. This visible/infrared separation device separates a visible/infrared composite image generated by a visible/near-infrared multi-wavelength camera into a visible image and an infrared image.

[How to generate attribute information]
In the above-described embodiment, the likelihood is obtained only from the visible image of the current frame, but the present invention is not limited to this. That is, the likelihood may also be obtained for the visible image of the past frame. For example, the attribute information generating means averages the likelihoods in a period that is a certain amount of time before the current time, and selects the combination of the tip of the sword and the player that has the highest average likelihood.

[Other Modifications]
Although fencing has been described as an example in the above-described embodiments, the present invention is not limited to this. That is, the present invention can also be applied to sports such as tennis, badminton, and volleyball, in which the positions of players do not change. For example, in the case of badminton, the object tracking device identifies the direction of the racket held by the player, associates the left and right players with the racket trajectory, and draws the racket trajectories held by both players in different colors.

Furthermore, the invention can also be applied to sports in which the positions of the players are swapped if the trajectories are not drawn in different colors. For example, an object tracking device can track a badminton shuttlecock and draw its trajectory. Furthermore, the present invention can also be applied to kendo and naginata. In addition, the present invention can draw the trajectory of a baton in an orchestra, or the trajectory of a sword or the like in a drama or movie.

In the above-described embodiment, the classifier is learned by SVM, but the present invention is not limited to this. For example, the discriminator can be learned by a neural network such as a recurrent neural network (RNN: Recurrent Neural Network), CRF (Conditional Random Fields), or the like. Moreover, in the present invention, not only a regression model discriminator but also a classification model discriminator can be used.

In the above-described embodiment, 18 joint points are detected, but all joint points may not be detected. The joint points of the upper body are considered to have a high correlation with the posture of the person, and particularly the joint points of the head and arms, so these joint points should be detected.
Further, although the method described in reference 1 is applied to detect joint points, the present invention is not limited to this. If a technique that can detect only the joint points of players is applied, the object tracking device does not need to include the person selection means.

In the above-described embodiment, the trajectory drawing means draws the trajectory, but the present invention is not limited to this. For example, the object tracking device may generate trajectory data indicating the trajectory of the object and output the generated trajectory data to the outside.

Although the object tracking device has been described as independent hardware in the above-described embodiments, the present invention is not limited to this. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk provided in a computer to operate cooperatively as the object tracking device described above. These programs may be distributed via a communication line, or may be distributed after being written in a recording medium such as a CD-ROM or flash memory.

In order to verify the effectiveness of the object tracking system in improving identification accuracy, we conducted an experiment by inputting a video of a fencing match into the object tracking system shown in FIG.
In the conventional method, tracking processing was performed only at the position of the tip of the sword using a particle filter, so erroneous tracking such as trajectory replacement occurred frequently. In this example, a video sequence in which mistracking occurred by the conventional method was used, and the accuracy (%) when identifying the left and right players for each video sequence was calculated. Furthermore, the processing speed (fps: frames/second) of the object tracking device was also measured.

The experimental results are shown in Table 1 below. A high accuracy of 97.6% was obtained on average for 5 video sequences. In this example, a video sequence in which mistracking occurred by the conventional method was used, but it was found that mistracking could be reduced in all video sequences by considering the pose of the person. Also, in this example, the average processing speed was about 2.8 fps, which was found to be practically sufficient. For example, real-time processing can be realized by using a GPU (Graphics Processing Unit) or performing identification processing in seconds.

FIG. 14 shows an image in which the likelihood distribution is visualized in order to verify the experimental results. As described above, the discriminator (SVM regression) can calculate the likelihoods of the left and right players, respectively. Therefore, in this embodiment, the likelihoods of the left and right players are calculated for all pixels of the image, and the calculated values are visualized in the form of a heat map. In FIG. 14, the brightness corresponding to the likelihood value indicates the player on the left side in red and the player on the right side in green. Furthermore, in FIG. 14, the detection position of the tip of the sword is illustrated with a circle, and the likelihood at that position is indicated numerically.

Furthermore, similar to FIG. 14, FIG. 15 shows the visualization of the likelihood distribution with another four images. As shown in FIGS. 14 and 15, the closer the players are to each other, the narrower the range of the likelihood distribution becomes, but the object tracking device can correctly identify the left and right players. Thus, it was found that the object tracking device can identify the player with high accuracy by learning the relationship between the position of the tip of the sword and the joint positions of the person.

1 Object Tracking System 10 Infrared Light Projector 20 Visible/Infrared Coaxial Light Camera 30 Object Tracking Device 31 Infrared Light Detection Means 33 Human Posture Acquisition Means 331 Human Posture Detection Means (Joint Position Detection Means)
333 person selection means 35 object identification means 351 feature vector calculation means 353 attribute information generation means 37 object tracking means 371 visible image storage means 373 trajectory drawing means (trajectory generation means)

Claims (6)

Using an infrared image obtained by photographing an infrared light marker attached to each moving object using infrared light and a visible image obtained by photographing a person moving each of the objects using visible light, the object An object tracking device for tracking
infrared light detection means for detecting the position of the infrared light marker from the infrared image as the position of the object;
joint position detection means for detecting each joint position of the person from the visible image;
feature vector calculation means for calculating a feature vector from the position of the object to each joint position;
The person corresponding to the object is selected by the feature vector using a classifier that has previously learned the relationship between the position of the object and the positions of the joints, and attribute information indicating the correspondence relationship between the object and the person. an attribute information generating means for generating
a trajectory generating means for generating a trajectory of the object based on the position of the object and the attribute information;
An object tracking device comprising: The attribute information generating means is
learning the classifier of the regression model for each person;
Using the classifier learned for each person, the likelihood of each combination of the object and the person is calculated from the feature vector, and the combination of the object and the person with the highest calculated likelihood is selected. 2. The object tracking device according to claim 1, wherein the attribute information is generated by selecting. The joint position detection means detects joint positions of all persons included in the visible image,
Based on the joint positions detected by the joint position detecting means, a body length from the neck joint to the hip joint is calculated for each person, and a predetermined number of the people are selected in descending order of the calculated body length. 3. The object tracking apparatus according to claim 1, further comprising: person selection means for outputting said joint positions of said selected person to said feature vector calculation means. 4. The object tracking device according to claim 3, wherein said feature vector calculating means normalizes said feature vector by said torso length. 5. The object tracking device according to claim 1, wherein the joint position detecting means detects at least the joint positions of the head and arms of the person. A program for causing a computer to function as the object tracking device according to any one of claims 1 to 5.

Images