JP2019101892A - Object tracking device and program thereof - Google Patents

Abstract

To provide an object tracking device capable of exactly tracking an object even when missing of a detection position is generated.SOLUTION: An object tracking device 30 is provided with: an infrared light detection unit 311 which detects an infrared light region; a position data generation unit 313 which generates position data of the infrared light region; a missing section specification unit 331 which specifies a missing section of the position data; a feature point detection unit 333 which detects a feature point of a visible image on the basis of the position data on both ends of the missing section; a position data interpolation unit 335 which interpolates the missing position data at a detection position of the feature point; and a trajectory drawing unit 353 which draws a trajectory on the basis of the position data.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an object tracking device for tracking an object using an infrared image and a visible image, and a program thereof.

  In recent years, with the development of video analysis technology, various applications using cameras have been proposed. The development of this technology is particularly remarkable in video analysis of sports scenes. For example, the Hawkeye system of tennis, which is also used in Wimbledon, three-dimensionally tracks a tennis ball using images of a plurality of fixed cameras, and performs IN / OUT determination. Also, in 2014 FIFA World Cup, called the goal line technology, it analyzes the images of several fixed cameras and automates the determination of goals. There is also known a TRACAB system that installs a large number of stereo cameras at a football stadium and tracks all players in the field in real time.

  These video analysis techniques are often based on the use of video taken with a camera having a time resolution of 30 frames per second (fps). For example, when shooting an object moving at a high speed that makes it difficult to visually recognize, such as a fencing sword tip or a badminton shuttle, extreme motion blur occurs in the object on the image (symbol α in FIG. 8). For this reason, it is extremely difficult to accurately measure the object position only from the image. When shooting objects that move at high speed, motion blur can be reduced by using a high-speed camera that exceeds 30 fps or by increasing the shutter speed. On the other hand, high-speed cameras are expensive, and there is a problem that increasing the shutter speed reduces the brightness of the image.

  Therefore, techniques have been proposed for attaching a sensor to an object and analyzing its movement. For example, in the inventions described in Patent Documents 1 and 2, the content of play is identified by machine learning from an acceleration sensor attached to a player's wear or a wristband. According to this conventional technique, it is possible to automatically determine the play content such as serve and volley in tennis. However, in the inventions described in Patent Documents 1 and 2, the sensor attached to the player often interferes with the game, and it is not practical to use the sensor in an actual game.

  Therefore, techniques have also been proposed that do not attach a sensor to an object. For example, according to the invention described in Patent Document 3, a reflective tape is attached to the tip of a fencing, infrared light is irradiated to the reflective tape, and the reflected light is detected from an infrared image to be accurate (robust) It tracks objects.

JP, 2014-187481, A JP, 2016-97228, A JP, 2017-156796, A

  However, in the invention described in Patent Document 3, when the object moves at a high speed or when the reflective tape does not face the camera, the reflected light from the reflective tape is weak, making it difficult to detect the object in the infrared image. As a result, the detection position is lost. Even if the reflected light becomes a sufficient amount of light and tracking can be resumed, if the missing section of the detection position is unnatural for a long time, it is difficult to represent the accurate trajectory of the object.

  Then, this invention makes it a subject to provide the object tracking apparatus which can track an object correctly, even when a detection position is missing, and its program.

  In view of the problems described above, an object tracking device according to the present invention is an infrared image obtained by photographing one or more objects moving with an infrared light marker by infrared light, and a visible light obtained by photographing the object by visible light An object tracking apparatus for tracking the object using an image, comprising: an infrared light detection unit, a position data generation unit, a missing section identification unit, a feature point detection unit, a position data interpolation unit, an object And a tracking unit.

According to this configuration, the object tracking device detects the region of the infrared light marker from the infrared image as an infrared light region by the infrared light detection unit.
Further, when the object tracking device can detect the infrared light region by the position data generation unit, the detection position of the infrared light region or the absence of the detection position when the infrared light region can not be detected; The position data including the time information of the infrared image which is the detection target of the infrared light region is generated.

  In addition, the object tracking device identifies the missing section of the position data by the missing section identifying unit, and determines whether the time length from the start time to the end time of the identified missing section is equal to or greater than a preset threshold. judge.

  Here, in the infrared image immediately before the start time and immediately after the end time, detection of the infrared light marker attached to the object is successful. Therefore, in the visible image of the start time and the end time, there is a high possibility that the object exists around the detection position in the infrared image.

  Therefore, the object tracking device acquires the detection position from the position data immediately before the start time and immediately after the end time when the feature point detection unit determines that the time length of the missing section is equal to or more than the threshold. An image area is set to the visible image of the start time and the end time based on the position, and a feature point is detected from the set image area.

Further, the object tracking device interpolates the position data of the start time and the end time by the position data interpolation unit at the detection position of the feature point, and outputs the interpolated position data to the missing section identification unit.
Further, the object tracking device tracks the object based on the position data when the time length of the missing section is less than the threshold value by the object tracking unit.
Then, the object tracking device specifies the missing section of the position data input from the position data interpolating unit by the missing section specifying unit.

  As described above, the object tracking device does not detect feature points in order of time, but detects feature points from visible images of start time and end time where there is a high possibility that an object exists, and detects the feature points Interpolate position data. Then, the object tracking device repeats the above process so as to narrow the missing section of the position data from both ends.

According to the present invention, the following excellent effects can be obtained.
The object tracking device according to the present invention detects feature points from visible images of start time and end time where an object is likely to exist, and interpolates position data at the detected positions of the feature points. Thus, the object tracking device can accurately track the object even if the detected position is missing.

It is a schematic block diagram of the object tracking system concerning the embodiment of the present invention. It is an enlarged view of a sword point. It is a block diagram which shows the structure of the object tracking apparatus of FIG. (A) And (b) is explanatory drawing explaining position data. (A) And (b) is explanatory drawing explaining a FAST feature-value. It is explanatory drawing explaining an example of a locus | trajectory synthetic | combination image. It is a flowchart which shows operation | movement of the object tracking apparatus of FIG. It is explanatory drawing explaining the motion blur in the image | video of fencing.

Outline of Object Tracking System
An outline of an object tracking system 1 according to an embodiment of the present invention will be described with reference to FIG.
In the following embodiments, in fencing, a sword tip (object) held by a player is described as a tracking target. During fencing, both players often move at high speed.

The object tracking system 1 tracks two sword tips moving at high speed using a visible / infrared coaxial camera 20 capable of photographing visible light and infrared light with the same optical axis, and its trajectory C (C ₁ , C ₂ ) is drawn. As shown in FIG. 1, the object tracking system 1 includes an infrared light projector 10, a visible / infrared coaxial camera 20, and an object tracking device 30.

The infrared light projector 10 is a general light projector which projects infrared light.
As shown in FIG. 2, the infrared light emitted by the infrared light projector 10 is reflected by a reflective tape (infrared light marker) 91 attached to the tips 90 of both players, and visible and infrared light described later Photographed by the optical axis camera 20.

  The reflective tape 91 is for reflecting the infrared light from the infrared light projector 10. One or more reflective tape 91 may be attached to the point 90, and the size and the number thereof are not particularly limited. In the example of FIG. 2, the point 90 has attached one rectangular reflective tape 91 on its side surface. Here, as for the point 90, one reflective tape 91 may be added to the side opposite to the side, and a band-like reflective tape 91 may be wound around the side (not shown).

  The visible and infrared coaxial camera 20 captures visible light and infrared light with the same optical axis, and generates a visible image I and an infrared image H with the same number of pixels. In the present embodiment, the visible / infrared coaxial camera 20 generates a visible image I obtained by shooting a fencing competition and an infrared image H obtained by shooting the reflective tape 91 of the point 90. Here, since the image coordinates of the sword tip 90 of the visible image I and the reflection tape 91 of the infrared image H correspond to each other, the trajectory C can be drawn without performing viewpoint conversion in a three-dimensional space.

The object tracking device 30 tracks the sword tips 90 of both players using the infrared image H and the visible image I input from the visible / infrared coaxial camera 20. The object tracking device 30 draws a locus C _1, C ₂ of the loop-taker point 90 of the two players tracked in different colors, a locus C _1, C ₂ drawn by synthesizing the visual image I, the locus-synthesized image Generate F.
In FIG. 1, the locus C _{1 of the} sword tip 90 possessed by the left player is illustrated by a broken line, and the locus C _{2 of the} sword tip 90 possessed by the right player is illustrated by an alternate long and short dash line.

[Configuration of Object Tracking Device]
The configuration of the object tracking device 30 will be described with reference to FIG.
As shown in FIG. 3, the object tracking device 30 includes an infrared light detection means 31, a visible image analysis means 33, and a time code linked CG synthesis means 35. Hereinafter, the time code linked CG synthesis means 35 will be abbreviated as the TC linked CG synthesis means 35.

Here, the object tracking device 30 receives the infrared image H and the visible image I of the frames 1,..., T−1, t,. We will process in order. Thereafter, the current frame (present frame) and t, the infrared image H and the visible image I of the current frame t and infrared image H _t and the visible image I _t. The subscript t indicates time information (for example, a time code) of each image or position information.

Infrared light detecting means 31 detects the loop-taker point 90 from the infrared image H _t of the current frame t, and generates the position data of the point of a sword 90. In the present embodiment, infrared light detecting means 31 detects the two hook point 90 included in the infrared image H _t, to generate position data for each point of a sword 90. The infrared light detection unit 31 includes an infrared light detection unit 311 and a position data generation unit 313.

Infrared detection unit 311 detects a region of the infrared light marker as infrared light region from the input from the visible and infrared same optical axis camera 20 infrared image H _t. The infrared light detection unit 311 can detect an infrared light region by any method. Hereinafter, an example of a detection method of the infrared light region in the infrared light detection unit 311 will be described.

In this embodiment, it extracts an area of the reflective tape 91 as a candidate blob from the infrared image H _t. Then, the infrared light detection unit 311 detects candidate blobs within the range of the area and shape feature amount set in advance as detection blobs in the order of the area of the candidate blob being wide by the upper limit number of objects set in advance. In the present embodiment, the upper limit number of objects is set to “2” which is the same number as that of the sword tip 90.

First, the infrared light detecting unit 311, using Equation (1) below, generates a binary infrared difference image and the infrared image H _t, of the previous frame and the infrared image H _t-1 by, to extract only the area Q _t of the moving object. In other words, the infrared light detecting section 311, an infrared image _{H t} of pixel _(x t, _{y t)} and the luminance value ^{H xy} _t of infrared image _{H t-1} of the pixel _{(x t-1, y} t- The difference between ₁ ) and the luminance value H ^xy _t-1 is determined. Then, the infrared light detection unit 311 extracts the area Q ^xy _t of the moving object whose _calculated difference exceeds the predetermined threshold R _bri2 as a candidate blob.

Here, x and y represent horizontal and vertical image coordinates. Further, the threshold R _bri2 is set in advance as an arbitrary value. Further, '0' and '255' in the equation (1) represent the luminance value of each pixel.
The infrared light detection unit 311 generates the binary infrared difference image Q ^xy _t in order to suppress the generation of the stationary noise blob, but extracts the region with high luminance in the infrared image H _t as the candidate blob You may

  Next, the infrared light detection unit 311 performs a morphology process (opening morphology process) on the extracted candidate blob to erase the noise blob in the small area. The morphological processing is processing for erasing noise blobs in a small area by inter-image calculation of an original image and an image group in which an image is shifted in pixel units in several directions.

  Next, the infrared light detection unit 311 performs a labeling process on candidate blobs remaining in the morphological process. This labeling process is a process of assigning a label (number) to a candidate blob.

  Next, the infrared light detection unit 311 obtains the position, area, and shape feature of the candidate blob subjected to the labeling process. Here, the position of the candidate blob is the center position or barycentric position of the candidate blob. Further, the shape feature amount of the candidate blob is assumed to be the circularity or the aspect ratio of the circumscribed rectangle.

  Then, the infrared light detection unit 311 deletes candidate blobs that are not within the range from the minimum area to the maximum area set in advance. Furthermore, the infrared light detection unit 311 deletes candidate blobs whose shape feature amount is not within a preset range. Furthermore, when the number of candidate blobs exceeds the upper limit number of objects, the infrared light detection unit 311 leaves two candidate blobs having large areas as detection blobs, and deletes other candidate blobs.

When the infrared light region can be detected (when the detection blob remains), the infrared light detection unit 311 detects the position of the detection blob in the infrared light region in association with the time information of the infrared image H _t The position is output to the position data generation unit 313 as a position.
On the other hand, when the infrared light region can not be detected (when the detection blob does not remain), the infrared light detection unit 311 detects that the infrared light region can not be detected together with the time information of the infrared image H _t Notify 313 (detection failure notification).

Position data generating unit 313, a detection position or detection failure notification infrared region input from the infrared light detecting section 311, based on the time information of the infrared image H _t, generates position data P _t It is a thing. Then, the position data generation unit 313 outputs the generated position data P to the missing section identification unit 331.

<Position data>
The position data will be described in detail below.
The position data P _t is information in which position information of the infrared light region is associated with time information (time code) of the infrared image H _t which is a detection target of the infrared light region.
In the present embodiment, the position information of the infrared light region indicates the detection position of the point 90. Here, when the detection position of the infrared light region is input from the infrared light detection unit 311, the position information of the infrared light region becomes image coordinates (x _t , y _t ) of the infrared light region. On the other hand, when the detection failure notification is input from the infrared light detection unit 311, the position information of the infrared light region becomes image coordinates (−1, −1) indicating a missing detection position.

As shown in FIG. 4 (a), each row of position data P is represents a time code of the infrared image H _t, and the positional information of the infrared region included in the infrared image H _t. In the position data P of FIG. 4A, the first to fourth lines and the tenth to eleventh lines indicate the detection position of the infrared light region, and the fifth to ninth lines indicate the missing detection position. Represent.

  For example, in the first line of the position data P, the time code of the infrared image H is “10: 25: 30.25”, and the image coordinates of the infrared light region are (1250, 350). In the fifth line of the position data P, the time code of the infrared image H is “10: 25: 30.29”, and the image coordinates of the infrared light region are (−1, −1). Represents the lack of

  Position data P is generated for each object. In the present embodiment, since there are two sword tips 90, position data P in FIG. 4 corresponds to the sword tip 90 of the left player, and position data P corresponding to the sword tip 90 of the right player is separately generated.

Returning to FIG. 3, the description of the configuration of the object tracking device 30 will be continued.
The visible image analysis means 33 specifies a missing section included in the position data P, detects feature points from visible images at both ends of the missing section, and interpolates the position data P. The visible image analysis unit 33 includes a missing section identification unit 331, a feature point detection unit 333, and a position data interpolation unit 335.

  The missing section identifying unit 331 identifies the missing section of the position data P input from the position data generating unit 313 or the position data interpolating unit 335, and determines whether the time length d of the identified missing section is equal to or greater than the threshold value m. It is to judge.

  First, the missing section identification unit 331 refers to each row of the position data P and identifies a missing section in which position information is continuously missing. In the present embodiment, the missing section identification unit 331 identifies a line including the image coordinates (−1, −1) in the position data P.

Next, the missing section identification unit 331 sets the earliest time as the start time T _ST of the missing section among the missing sections in which the image coordinates (−1, −1) continue, and the latest time is the ending time of the missing section _Calculated as T _ED . Then, the missing segment identification unit 331 obtains a time length d from the start time T _ST of the identified missing segment to the end time T _ED .

Next, the missing section identification unit 331 determines whether or not the calculated time length d of the missing section is equal to or greater than the threshold value m. This threshold m is preset with an arbitrary value (for example, '3' frame).
When the time length d is equal to or more than the threshold value m, the missing section identification unit 331 outputs the position data P and the start time T _ST and the end time T _ED of the missing section obtained to the feature point detection section 333.
On the other hand, when the time length d is less than the threshold value m, the missing section identification unit 331 writes the position data P into the position data storage unit 351.

When the time length d of the missing section is equal to or greater than the threshold value m, the feature point detection unit 333 acquires detection positions immediately before the start time and immediately after the end time from the position data P, and starts and ends the start time based on the acquired detection positions. It sets the image area to the time of the visual image I _t, and detects a feature point from an image region set. The feature point detection unit 333, feature point detection unit 333, the position data P from the missing section identifying unit 331 is input, a visible image I _t is inputted from the visible and infrared same optical axis camera 20.

First, the feature point detection unit 333 obtains a start just before time _{T ST-1} from the start time _{T ST} missing sections. The time immediately before the start _TST-1 represents a time (time code) immediately before the start time _TST . In addition, the feature point detection unit 333 obtains time T _{ED + 1} immediately after the end from the end time T _ED of the missing section. The time immediately after the end _{TED + 1} represents the time (time code) one after the end time _TED .

Here, in the infrared image H immediately before the start time _TST-1 and immediately after the time end _{TED + 1} , the detection of the point 90 has succeeded. Therefore, in the visible image _{I ST} start time _{T ST,} there is a high possibility that the loop-taker point 90 is present around the detection position immediately before the start time _{T ST-1 (x ST-} 1, y ST-1). Similarly, also in the visible image _{I ED} end time _{T ED,} it is likely that there is a point of a sword 90 around the detection position of the end immediately after time _{T ED + 1 (x ED +} 1, y ED + 1).

Therefore, the feature point detection unit 333, immediately before the start time _T detected position from the position data _{P ST-1} of the _{ST-1 (x ST-1} , y ST-1) acquires, the detection position _{(x ST-1,} An image area (search area) based on y _ST-1 ) is set as the visible image I _ST . This image area can be set with an arbitrary shape and size centered on the detection position, and is, for example, a rectangular area of 160 vertical pixels × 160 horizontal pixels.
The feature point detection unit 333 obtains the detected position from the position data _{P ED + 1} of immediately after the end time _{T ED + 1 (x ED +} 1, y ED + 1), an image area the detected position _{(x ED + 1, y ED} + 1) as a reference Set to visible image I _ED .

Then, feature point detection unit 333 in the image area, using the equation (2) below, a visible image I _t, of the previous frame difference image S _t of a visible image I _t-1 Generate That is, the feature point detection unit 333, the luminance value ^{I xy} _t of a pixel of the visible image _{I t (x t, y t} ), the visible image _{I t-1} pixels _{(x t-1, y t} -1) A difference image S ^xy _{t in} which the difference from the luminance value I ^xy _t-1 exceeds a preset threshold R _bri is extracted.

In addition, when applying Formula (2) to visible image _IST , it is set as t = ST, and when Formula (2) is applied to visible image _IED , it is set as t = ED. In addition, the threshold value R _bri is set in advance as an arbitrary value.
The feature point detection unit 333, similarly to the infrared light detecting section 311, subjected to opening morphological processing to the differential image S _t, may be removed Noizuburobu subregion.

By detecting the feature point from the difference image S _t, the case of a fixed camera image, it is possible to suppress the occurrence of Noizuburobu stationary. For moving the camera image, since the difference due to the movement of the camera occurs, the effect is reduced to utilize the difference image S _t. In that case, the feature point detection unit 333 may create an edge image and detect feature points from an image area having a high edge value. Furthermore, the feature point detection unit 333 may detect a feature point using the difference value, the edge value, or the luminance value as it is without performing binarization.

Next, the feature point detection unit 333 detects feature points from the difference image S ^xy _ST set in the visible image I _ST , and among the detected feature points, the infrared light region at the start time T _ST-1 immediately before the start The feature point closest to the detection position (x _ST-1 , y _ST-1 ) is acquired as the feature point of the start time T _ST .
Furthermore, the feature point detection unit 333 detects a feature point from the difference image S ^xy _ED set in the visible image I _ED and detects the detection position of the infrared light region at time T _{ED + 1} immediately after the end from the detected feature points ( The feature point closest to x _{ED + 1} , y _{ED + 1} ) is acquired as the feature point of the end time T _ED .
Thereafter, the feature point detection unit 333 outputs the position of the feature point of the acquired start time T _ST and end time T _ED to the position data interpolation unit 335.

  As a method of detecting the feature points described above, Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), etc. are representative, but the feature point detection unit 333 uses any method of detecting feature points. May be In this embodiment, it is assumed that the FAST (Features from Accelerated Segment Test) feature described in the following reference is used.

  Reference: Edward Rosten and Tom Drummond, "Machine learning for high-speed corner detection," In Proc. Of the European Conference on Computer Vision (ECC V 2006), pp. 430-443, 2006.

<FAST feature amount>
This FAST feature value targets only the corner as a feature point, and quickly and efficiently detects the feature point by the decision tree. Here, the detection of the FAST feature amount from the visible image I of FIG. 5A will be described. As shown in FIG. 5B, in the FAST feature amount, 16 pixels around the pixel of interest p are observed. Then, in the FAST feature amount, when n or more pixel values are successively brighter or darker than the threshold value t among the 16 observation pixels in comparison with the pixel value of the target pixel p, the target pixel p Is detected as a corner (where n is a natural number of 1 or more).
In FIG. 5B, the target pixel of the visible image I is shown by p, and the observation pixel is shown by a numerical value of 1-16.

The FAST feature uses a decision tree to quickly and efficiently detect highly reproducible feature points. As shown in Equation (3) below, the 16 observation pixels are classified into three values: brighter, similar, or darker.
In Equation (3), I _p represents the luminance value of the target pixel p, x represents the position of the observation pixel, I _{p → x} represents the luminance value of the observation pixel, and t represents the threshold value.

  An observation pixel classified into three values is regarded as a feature vector, and is detected as a corner when the condition that n or more pixel values of the observation pixel are brighter or darker successively is satisfied. On the other hand, when this condition is not satisfied, it is treated as a non-corner. The process of constructing a decision tree in this way and selecting an observation pixel on a circle capable of optimally classifying corners and non-corners as branch nodes is performed recursively to obtain a whole tree structure.

  In many feature point detection methods, only the effective pixels are detected among adjacently detected feature points by extracting the local maximum value of the response value representing the feature point likeness. However, since the response value is not extracted in the FAST feature amount, the response value V is calculated by the following equation (4). By setting a pixel having the highest response value V as a corner point among the corner points detected adjacent to each other, corner points separated from each other can be detected.

Returning to FIG. 3, the description of the configuration of the object tracking device 30 will be continued.
The position data interpolation unit 335 interpolates the position data P of the start time _TST and the end time _TED with the detection position of the feature point input from the feature point detection unit 333. That is, the position data P where the detection position is missing is updated with the position of the feature point detected by the feature point detection unit 333.

Subsequently, the position data interpolation unit 335 outputs the interpolated position data P to the missing section identification unit 331.
Then, the missing segment identification unit 331 identifies the missing segment again from the position data P input from the position data interpolating unit 335. Thereafter, the visible image analysis means 33 repeats the above process.

<Interpolation of position data>
The interpolation of the position data by the visible image analysis means 33 will be described as a specific example with reference to FIG. 4 (see FIG. 3 as needed).
Here, it is assumed that the position data P of FIG. 4A is input, and the threshold value m is a '3' frame.

First, the first interpolation in the position data P of FIG. 4A will be described.
The missing section identification unit 331 refers to each line of the position data P, and identifies the fifth to ninth lines in which the image coordinates (−1, −1) are continuous as a missing section. Then, the missing section identification unit 331 sets the earliest time “10: 25: 30.29” among the missing sections on the fifth to ninth lines as the start time T _ST and the latest time “10:25: 31.03 "seek as the end time _{T ED.}

Next, the missing segment identification unit 331 sets the time length d from the start time T _ST "10: 25: 30.29" to the ending time T _ED "10: 25: 31.03" of the missing segment to a '5' frame. The threshold determination of the time length d of the missing section and the threshold value m is performed. Here, since the time length d = '5' is equal to or more than the threshold value m = '3', the missing section identification unit 331 determines that the position data P and the start time T _ST "10: 25: 30.29" and the end time T _ED It outputs “10: 25: 31.03” to the feature point detection unit 333.

Feature point detector 333, the start time _{T ST} starts immediately before time "10: 25: 30.29" is one frame before _{T ST-1} "10: 25: 30.28" is calculated. The feature point detection unit 333, end time _{T ED} "10: 25: 31.03" is after 1 frame immediately after the end time _{T ED + 1} "10: 25: 31.04" is calculated. Then, the feature point detection unit 333 obtains image coordinates (1255, 344) from the position data P at the time immediately before the start TST _-1 , and obtains image coordinates (1265, 331) from the position data P at the time TED _{+ 1} immediately after the end. get.

Next, the feature point detection unit 333 sets an image area centered on the image coordinates (1255, 344) in the visible image I _ST at the start time T _ST , and detects feature points from this image area. Further, the feature point detecting unit 333 sets an image area obtained by image coordinates (1265,331) centered on the visible image _{I ED} end time _{T ED,} detects feature points from the image area.

The position data interpolation unit 335 updates the image coordinates (−1, −1) of the start time T _ST with the image coordinates (1256, 343) of the feature point in the position data P of FIG. 4A. Further, the position data interpolation unit 335 updates the image coordinates (−1, −1) of the end time T _ED with the image coordinates (1264, 331) of the feature point.

Subsequently, the second interpolation in the position data P of FIG. 4B will be described.
The missing section identification unit 331 refers to each line of the position data P and identifies the sixth to eighth lines in which the image coordinates (−1, −1) are continuous as a missing section, and the start time T _ST “ 10: 25: 31.00 "and end time T _ED " 10: 25: 31.02 "are calculated | required.

Next, because the time length d of the missing section d = '3' is equal to or greater than the threshold value m = '3', the missing section identification unit 331 ends the position data P and the start time T _ST "10: 25: 31.00" and ends The time T _ED “10: 25: 31.02” is output to the feature point detection unit 333.

The feature point detection unit 333 calculates a time immediately before the start T _ST-1 “10: 25: 30.29” and a time immediately after the end T _{ED + 1} “10: 25: 31.03”. Then, the feature point detection unit 333 obtains image coordinates (1256, 343) from the position data P at the time immediately before the start TST _-1 , and obtains image coordinates (1264, 331) from the position data P at the time TED _{+ 1} immediately after the end. get.

Next, the feature point detection unit 333 sets an image area centered on the image coordinates (1256, 343) in the visible image I _ST at the start time T _ST , and detects feature points from this image area. Further, the feature point detecting unit 333 sets an image area obtained by image coordinates (1264,331) centered on the visible image _{I ED} end time _{T ED,} detects feature points from the image area.

The position data interpolator 335 detects the image coordinates (-1, -1) of the start time T _ST and the end time T _ED in the position data P of FIG. Update in image coordinates.

Thus, a visible image analysis means 33, the visible image I _ED visual image I _ST and end time T _ED detectable starting time T _ST objects with high probability to detect an object, gradually the position data P By narrowing the missing section, accurate position data P can be generated.

Although it has been described that the position data P in FIG. 4 has one missing section, the position data P may include a plurality of missing sections. In this case, the visible image analysis means 33 may apply the above-described processing to each of the missing sections.
Moreover, it goes without saying that the position data is not limited to the example of FIG.

Returning to FIG. 3, the description of the configuration of the object tracking device 30 will be continued.
TC linked CG synthesizing unit 35 as a trigger time code T _t of the visible image I _t input from the visible and infrared same optical axis camera 20, from the stored position data P to obtain the position coordinates of the object trajectory draw a, it is to synthesize drawing trajectory into a visible image I _t.

  Usually, a slow VTR for playback records live images and CG images generated in real time on the VTR and rewinds them at the timing of use and then reproduces them. However, when a CG image can not be generated in real time, it is necessary to reproduce the visible image I recorded in the VTR and draw a locus after interpolating the position data P. In this case, a certain working time is required and the efficiency is lost. Therefore, it is preferable to draw a locus at timing synchronized with the reproduction of the video recorded in the VTR and synthesize CG on the spot.

As shown in FIG. 3, the TC-linked CG synthesis unit 35 includes a position data storage unit 351 and a trajectory drawing unit (object tracking unit) 353.
The position data storage unit 351 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a memory that stores position data P. The position data P is written by the missing section identification unit 331 and referred to by the trajectory drawing unit 353.

Locus drawing unit 353 reads the position data P _t corresponding to the time information of the visual image I _t input from the visible and infrared same optical axis camera 20 from the position data storage unit 351. Then, the locus drawing unit 353 draws the locus of the point 90 on the pixel (x _t , y _t ) of the read position data P _t , and externally synthesizes the locus synthetic image F _t obtained by synthesizing the drawn locus and the visible image I _t Output.

For example, the locus drawing unit 353, a locus _C 1, _{C 2} of the loop-taker point 90 of the two players drawn in different colors, a locus _C 1, _{C 2} drawn by synthesizing the visual image _{I t,} locus-synthesized image F Generate _t . Here, the position data P is interpolated even when the detection of the sword tip 90 of the left player fails, so the locus drawing unit 353 draws the locus C ₁ of the sword tip 90 of the left player as shown in FIG. it can.
In FIG. 6, the rectangle β represents the image area set by the feature point detection unit 333, and the circle γ represents the tip 90 detected by the infrared light detection unit 311.

[Operation of Object Tracking Device]
The operation of the object tracking device 30 will be described with reference to FIG. 7 (see FIG. 3 as needed).
As shown in FIG. 7, the infrared light detection unit 311 generates a binary infrared difference image, and performs morphological processing on the extracted candidate blobs (step S1).

The infrared light detection unit 311 performs labeling processing on the candidate blobs remaining in the morphological processing to obtain the position, area, and shape feature of the candidate blob. Then, the infrared light detection unit 31 performs filtering based on the area and shape feature amount to detect a detection blob (infrared light region) (step S2).
The position data generation unit 313 generates position data in which the detection position of the infrared light region is associated with the time information of the infrared image (step S3).

The missing section identification unit 331 acquires the start time T _ST and the end time T _ED of the missing section from the position data, and obtains the time length d from the start time T _ST to the end time T _ED (step S4).
The missing section identification unit 331 determines whether the time length d of the missing section is equal to or greater than the threshold value m (step S5).

When the time length d is equal to or more than the threshold value m (Yes in step S5), the feature point detection unit 333 obtains the detection positions of time _TST-1 immediately before the start and time _{TED + 1} immediately after the end from the position data (step S6). .
The feature point detection unit 333 sets, in the visible image, an image area based on the detection position of the time immediately before the start _TST-1 and the time immediately after the end _{TED + 1} (step S7).

The feature point detection unit 333 detects feature points in the image area set in the visible image (step S8).
The position data interpolation unit 335 interpolates position data at a feature point closest to the past detected position (step S9), and returns to the process of step S4.

If the time length d is less than the threshold value m (No in step S5), the missing section identification unit 331 writes the position data to the position data storage unit 351.
The trajectory drawing unit 353 reads from the position data storage unit 351 position data corresponding to time information of the visible image input from the visible / infrared coaxial camera 20 (step S10).
The locus drawing unit 353 draws the locus of the object according to the detected position of the read position data, and combines the drawn locus with the visible image (step S11).

[Operation / effect]
The object tracking device 30 according to the embodiment of the present invention detects feature points from the visible images of the start time T _ST and the end time T _ED with high possibility of the presence of the point 90, and detects position data at the detected positions of the feature points. Interpolate. Thereby, the object tracking device 30 can accurately draw the trajectory of the point 90 even when the detected position is missing.

  Furthermore, the object tracking device 30 can reliably track the moving tip 90 and draw an accurate trajectory without attaching a sensor to the tip 90. That is, the object tracking device 30 can visualize the movement of a high-speed moving object that is difficult to visually recognize with the naked eye or a normal camera image. The object tracking device 30 can express the movement of the fencing sword tip 90 moving at a speed that can not be confirmed with the naked eye in video. Thus, the object tracking device 30 can draw the trajectory of the point 90 in fencing and can improve the quality of the trajectory drawing.

As mentioned above, although the embodiment of the present invention has been described in detail, the present invention is not limited to the above-described embodiment, and includes design changes and the like within the scope of the present invention.
The position data interpolated by the object tracking device is not particularly limited in how to use the position data, and can be used other than the drawing of a trajectory.

  In the above embodiment, the object tracking device has been described as receiving a visible image and an infrared image from a visible / infrared coaxial camera, but the present invention is not limited to this. That is, the object tracking device may be disposed downstream of the other image processing device, and the visible image and the infrared image which failed in the tracking by the other image processing device may be input.

  In the above-described embodiment, fencing has been described as an example, but the application target of the object tracking device is not limited to this. That is, the object tracking device can be applied to sports such as tennis, badminton and volleyball.

  Furthermore, the object tracking device can also be applied to sports in which the player's position changes if the trajectory is not drawn in a different color. For example, the object tracking device can track a badminton shuttle and draw its trajectory. In addition, the object tracking device can draw a trajectory of a baton in an orchestra, and a trajectory of a sword in a drama or a movie.

  In the object tracking device, the moving speed of the object to be tracked is not particularly limited, and in particular, the moving speed at which motion blur occurs can be reliably tracked. This moving speed largely depends on the distance between the visible and near infrared coaxial camera and the object, the photographing angle of view and sensitivity of the visible and near infrared coaxial camera, and the amount of noise. For example, when the time resolution is 30 frames per second, motion blur increases when the moving speed exceeds 100 km / h.

  Although the object tracking device has been described as an independent hardware in the above embodiment, the present invention is not limited to this. For example, the object tracking device can also be realized by an object tracking program that causes hardware resources such as a CPU, a memory, a hard disk, and the like included in a computer to cooperate as the respective means described above. This program may be distributed via a communication line, or may be distributed by writing on a recording medium such as a CD-ROM or a flash memory.

  The present invention can be used for video analysis of sports scenes such as fencing. The present invention can also be used to produce movies, games and dramas. Furthermore, the invention can also be used in industrial and security systems.

DESCRIPTION OF SYMBOLS 1 object tracking system 10 infrared light projector 20 visible / infrared same optical axis camera 30 object tracking device 31 infrared light detection means 311 infrared light detection unit 313 position data generation unit 33 visible image analysis means 331 missing section identification unit 333 Feature point detection unit 335 Position data interpolation unit 35 Time code interlocking type CG synthesizing unit (TC interlocking type CG synthesizing unit)
351 Position Data Storage Unit 353 Trajectory Drawing Unit (Object Tracking Unit)
90 point 91 reflective tape (infrared marker)

Claims (4)

An object tracking device for tracking an object by using an infrared image obtained by capturing infrared light with one or more objects to be moved with an infrared light marker and a visible image obtained by capturing the object with visible light ,
An infrared light detection unit that detects an area of the infrared light marker from the infrared image as an infrared light area;
When the infrared light region can be detected, the detection position of the infrared light region or the infrared light which is the detection target of the infrared light region when the infrared light region can not be detected A position data generation unit that generates position data including time information of an image;
A missing section identification unit that identifies a missing section of the position data and determines whether the time length from the start time to the ending time of the identified missing section is equal to or greater than a preset threshold value;
When the time length of the missing section is equal to or more than the threshold value, the detection position is acquired from position data immediately before the start time and immediately after the end time, and the start time and the end of the image area based on the acquired detection position A feature point detection unit that sets a visible image of time and detects feature points from the set image area;
A position data interpolation unit that interpolates position data of the start time and the end time at a detection position of the feature point and outputs the interpolated position data to the missing section identification unit;
An object tracking unit configured to track the object based on the position data when the time length of the missing section is less than the threshold value;
The object tracking device, wherein the missing segment identification unit identifies a missing segment of the position data input from the position data interpolation unit. A position data storage unit for storing the position data;
The missing section identification unit writes the position data to the position data storage unit when the time length of the missing section is less than the threshold value;
The object tracking unit acquires position data corresponding to time information of the visible image from the position data storage unit, draws the locus of the object based on the acquired detected position of the position data, and draws the locus The object tracking device according to claim 1, further comprising: a trajectory drawing unit that synthesizes the image with the visible image.   The object tracking device according to claim 1, wherein the feature point detection unit detects a FAST feature amount from the image area.   An object tracking program for causing a computer to function as the object tracking device according to any one of claims 1 to 3.