JP2018206285A - Object tracking device and object trajectory generation system - Google Patents

Abstract

To provide an object tracking device capable of optimally tracking an object in which occlusion is frequently generated, with few stationary cameras.SOLUTION: An object tracking section 51 as the object tracking device includes: a tracking part 51b for tracking an object by a sequential learning-type tracking method; and a search range setup part 51c setting up a search range to frames of stationary cameras such that as a size of the object included in current frames is enlarged, the size of the object search range in next frames is enlarged. The tracking part 51b tracks the object within the search ranges set up in the frames by the search range setup part 51c.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a technique for tracking an object from a photographing result of a fixed camera.

  With the progress of video analysis technology, various applications using cameras as sensors have been proposed. The tennis hawkeye system also used in Wimbledon tracks a tennis ball three-dimensionally by using a plurality of fixed camera images as sensors, and performs in / out determination involving a judge. The goal line technology used in the 2014 FIFA World Cup automates goal determination by analyzing video from several fixed cameras. Furthermore, real-time video analysis technology in sports such as a tracking system (for example, TRACAB) that installs a large number of stereo cameras in a soccer stadium and tracks all players in the field in real time has been advanced.

  Patent Documents 1 and 2 describe a method of tracking an object moving in a wide area while avoiding an occlusion phenomenon that makes tracking difficult due to overlapping of object images from within a camera video. In these methods, an object is tracked from a large number of camera images of about eight cameras, and when occlusion occurs, the object is robustly tracked using information of other cameras in which no occlusion has occurred.

  Patent Document 3 describes a method of attaching a transmitter to a tracking target object such as a soccer ball and measuring the position in real time.

JP 2001-94975 A JP-A-2006-1447 JP-T-2006-504088

  However, the methods described in Patent Documents 1 and 2 are effective in an environment where a large number of cameras can be installed, but are difficult to use when the number of cameras is limited as in a curling stadium. In addition, the technique described in Patent Document 3 cannot be used when the processing of the equipment used for the competition is prohibited by rules, such as curling.

  The present invention has been made in view of the above points, and it is an object of the present invention to provide an object tracking device and an object trajectory generation system capable of suitably tracking an object in which occlusion frequently occurs with a small number of fixed cameras. And

  In order to solve the above-described problem, an object tracking device of the present invention is an object tracking device that tracks an object included in a fixed camera image, and includes a tracking unit and a search range setting unit.

  In the object tracking device, the tracking unit tracks the object by a sequential learning type tracking method, and the search range setting unit increases the size of the object included in the current frame of the fixed camera video as the next frame increases. The search range is set in the frame of the fixed camera video so that the size of the search range of the object at is large. The object tracking device tracks the object within the search range set in the frame by the search range setting unit by the tracking unit.

  The object tracking device may include an object candidate image generation unit. In this case, the object tracking device generates an object candidate image obtained by extracting the object candidates included in the frame based on the color of the object by the object candidate image generation unit, and the tracking unit generates the object candidate image. The object is tracked using a candidate image, and the search range is set in the object candidate image by the search range setting unit.

  The object trajectory generation system of the present invention includes the object tracking device, a projective transformation device, and a trajectory generation device. The object trajectory generation system performs projective conversion of the coordinates of the object tracked by the tracking unit by the projective conversion device, and the position group of the object tracked by the object tracking device by the trajectory generation device, or A trajectory of the object is generated based on a position group of the object tracked by the object tracking device and projectively transformed by the projective transformation device.

  The projective conversion device includes a first conversion unit and a second conversion unit. The projective transformation device performs projective transformation of the coordinates of the object included in the fixed camera image to the coordinate system for conversion by the first conversion unit, and the coordinate system for conversion by the second conversion unit. Projective transformation is performed on the coordinates of the object converted into the coordinate system from a viewpoint different from that of the fixed camera video.

  The projective transformation device transforms the coordinates of the object obtained by photographing the field from one side by the first conversion unit into the coordinate system for conversion when the field is viewed from above, and the second conversion unit. Thus, the coordinates of the object converted into the conversion coordinate system may be converted into a coordinate system in which the field is viewed from one side at a different angle of view.

  The projective transformation device converts the coordinates of the object obtained by photographing the field from one side to the coordinate system for conversion when the field is viewed from above by the first conversion unit. The coordinates of the object converted into the coordinate system for conversion by the conversion unit are rotated by 180 ° in the plane of the field, and the coordinates of the object rotated by 180 ° are the other opposite to the one side of the field. The configuration may be such that the coordinate system is viewed from the side.

  The object trajectory generation system may be configured to generate the trajectory of the object by spline interpolation by the trajectory generation device.

  The object may be a curling stone.

  According to the present invention, an object in which occlusion frequently occurs can be suitably tracked with a few fixed cameras.

It is a mimetic diagram showing an object locus generating system concerning an embodiment of the present invention. (A) is a figure which shows an example of a picked-up image, (b) is an enlarged view which shows the object vicinity of (a). (A) is a figure which shows an example of a picked-up image, (b) is an enlarged view which shows the object vicinity of (a). (A) is a figure which shows an example of a picked-up image, (b) is an enlarged view which shows the object vicinity of (a). It is a block diagram which shows the object locus | trajectory generation system which concerns on embodiment of this invention. FIG. 7 is a diagram for explaining a search range setting method, in which (a) is a diagram illustrating an example of an object and a search range in a t−1th object candidate image, and (b) is a tth object candidate image. FIG. 6C is a diagram illustrating an example of an object and a search range in FIG. 6C, and FIG. 8C is a diagram illustrating an example of an object and a search range in a t + 1th object candidate image. It is a figure for demonstrating the correspondence method to a view angle change, and the inversion method of a locus | trajectory, Comprising: (a) is a figure which shows an example of the imaging | photography result of a curling sheet by a fixed camera, (b) is a projection to an upper viewpoint coordinate system It is a figure which shows an example of the imaging | photography result of the converted curling sheet. It is a figure for demonstrating the response | compatibility method to a view angle change, Comprising: It is a figure which shows an example of the imaging | photography result of the curling sheet converted into the coordinate system of the fixed camera of another view angle. It is a figure for demonstrating the reversal method of a locus | trajectory, Comprising: (a) is a figure which shows an example which reversed the imaging | photography result of the curling sheet | seat by which projection conversion was carried out to the upper viewpoint coordinate system, (b) is a figure of the other fixed camera. It is a figure which shows an example of the imaging | photography result of the curling sheet converted into a coordinate system. It is a figure which shows an example of the synthesized image of the locus | trajectory of an object. It is a flowchart explaining the operation example of the object locus | trajectory generation system which concerns on embodiment of this invention. It is a flowchart explaining the operation example of the object locus | trajectory generation system which concerns on embodiment of this invention. It is a flowchart explaining the operation example of the object locus | trajectory generation system which concerns on embodiment of this invention.

  An embodiment of the present invention will be described in detail with reference to the drawings, taking as an example the case where the apparatus and apparatus of the present invention are applied to tracking of a curling stone as an object. In the following description, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted.

  Curling is a game in which stones are thrown into a circle called a house on the ice, and points are scored according to their positions. The ease of throwing stones depends greatly on the curling sheet condition of the day and the changes in the curling sheet condition caused by stone throwing. The trick is to create a situation where the enemy team is thrown to a course that is difficult for the enemy team to throw, while finding the easy-to-throw course on the spot.

  Although there is a limit to telling the state of the curling seat and the tactics of each team only with the camera image, the object trajectory generation system of the present invention displays the trajectory of the stones cumulatively to the viewer with commentary comments. Can convey the situation in an easy-to-understand manner. Stone trajectory display in curling competition is the first attempt, and the video expression itself is also novel.

As shown in FIG. 1, the object trajectory generation system 1 according to the embodiment of the present invention includes a curling sheet 2 having a curling length L _{A of} about 45 [m] and a width W _{A of} about 5 [m]. This is a system for displaying a trajectory of a stone as the object 4. Houses 3A and 3B are provided at both ends in the length direction of the curling sheet 2, respectively. The object 4 is thrown from the house 3B side to the house 3A at the odd end, and is thrown from the house 3A side to the house 3B at the even end. The object trajectory generation system 1 includes a first fixed camera 10A and a second fixed camera 10B as the fixed camera 10, a switcher 20, an operation unit 30, a display unit 40, a control unit 50, an inserter 60, Is provided.

<First fixed camera>
The first fixed camera 10A is for photographing the object 4 thrown toward the house 3A provided at one end of the curling sheet 2 in the length direction, and photographs the curling sheet 2 from the house 3A side. To do. The photographing result of the first fixed camera 10 </ b> A is output to the switcher 20.

  Here, as shown in FIGS. 2A to 4A, in the photographing result of the first fixed camera 10A, the object 4 moves from the back side (and the upper side) to the front side (and the lower side) of the screen. Move towards. Further, as shown in FIGS. 2B to 4B, the object 4 is occluded (hidden) by a curling player or the like.

<Second fixed camera>
As shown in FIG. 1, the second fixed camera 10 </ b> B is for photographing an object 4 thrown toward a house 3 </ b> B provided at the other longitudinal end of the curling sheet 2. The sheet 2 is photographed from the house 3B side. The imaging result of the second fixed camera 10B is output to the switcher 20.

<Third fixed camera>
In FIG. 1, a third fixed camera 10C is shown. Similarly to the first fixed camera 10A, the third fixed camera 10C is for photographing the object 4 thrown toward the house 3A provided at one end in the length direction of the curling sheet 2. The curling sheet 2 is photographed from the house 3A side. The photographing result of the third fixed camera 10 </ b> C is output to the switcher 20. That is, the first fixed camera 10A and the third fixed camera 10C capture the curling sheet 2 from the same direction at different angles of view. The third fixed camera 10C captures the curling sheet 2 adjacent to the curling sheet 2 photographed by the first fixed camera 10A, or the same curling as the curling sheet 2 photographed by the first fixed camera 10A. The sheet 2 is photographed at different angles of view on different days.

<Switcher>
The switcher 20 acquires the imaging results of the first fixed camera 10A and the second fixed camera 10B, and outputs one of the acquired imaging results to the control unit 50 and the inserter 60 based on the operation result by the user. In the present embodiment, the switcher 20 moves the object 4 from the back side to the near side among the shooting results of the fixed cameras 10A and 10B (the shooting result of the first fixed camera 10A at the odd end and the first at the even end. The imaging result (for example, see FIGS. 2 to 4) of the second fixed camera 10 </ b> B is output to the control unit 50 and the inserter 60.

<Operation section and display section>
The operation unit 30 is configured by a keyboard, a mouse, and the like, and outputs an operation result of the operation unit 30 by the user to the control unit 50. The display unit 40 is configured by a monitor or the like, and displays an image based on image data from the control unit 50.

<Control unit>
The control unit 50 includes a CPU (Central Processing Unit), a ROM (Read-Only Memory), a RAM (Random Access Memory), an input / output circuit, and the like. As illustrated in FIG. 5, the control unit 50 includes an object tracking unit 51, a projective conversion unit 52, a trajectory generation unit 53, a storage unit 54, and a trajectory output unit 55 as functional units.

<Object Tracking Unit (Object Tracking Device)>
The object tracking unit 51 tracks the object 4 included in the video captured by the fixed cameras 10A and 10B in the video. The object tracking unit 51 includes an object candidate image generation unit 51a, a tracking unit 51b, and a search range setting unit 51c.

<< Object candidate image generator >>
Objects candidate image generation unit 51a, an object fixed camera 10A, the fixed camera image I _t is a frame constituting the image captured by one of 10B obtained through the switcher 20, from the obtained fixed camera image I _t Candidate image _St is generated. Here, t is the number of a frame constituting the video. Objects candidate image S _t is an image candidate of the object 4 is within the search range to below 5 (see FIG. 6) is extracted.

First, the object candidate image generation unit 51a, the color of the object 4 (e.g., red or yellow) on the basis of the extracted ones object 4 and the same color as an object candidate from among the objects included in the fixed camera image I _t. Subsequently, the object candidate image generation unit 51a performs a process of filling the image area other than the extracted object candidate with black.

Subsequently, the object candidate image generation unit 51a causes the display unit 40 to display an image from which the object candidates are extracted. Subsequently, the object candidate image generation unit 51a, based on the operation result of the operation unit 30 by the operator who viewed the image displayed on the display unit 40, with respect to the first sheet of the fixed camera image I _t, the initial position of the object 4 as well as specify the sets initial search range 5 around the specified position of the object 4 (see FIG. 6) with respect to the first sheet of the fixed camera image I _t. The initial position of the object 4 specified here is for setting the initial search range 5. The initial size (vertical and horizontal dimensions) of the search range 5 is stored in advance.

Subsequently, the object candidate image generation unit 51a narrows down the object candidates by performing a labeling process on the image region in the search range 5 (see FIG. 6) and filtering based on the area and shape. Here, the object candidate image generation unit 51a determines that the area of the object candidate is larger than the preset maximum value, the area of the object candidate is smaller than the preset minimum value, and the shape of the object candidate is If the shape is different from the preset shape, the object candidate is excluded. The maximum value and the minimum value of the area of the object candidate are set based on the area of the object 4 located at both ends in the length direction of the curling sheet 2 and stored in advance. Subsequently, the object candidate image generation unit 51a performs a process of filling the object candidates excluded by the filtering with black. Thereby, the object candidate image _St is obtained.

Subsequently, the object candidate image generation unit 51a, with respect to the object candidate image S _t of the first sheet of the fixed camera image I _t, of the object candidates, the search range 5 objects 4 the closest to the center (see FIG. 6) And the size of the object 4 (for example, the length of one side) is stored.

Objects candidate image generation unit 51a, a second or subsequent object candidate image _{S t,} and outputs to the tracking unit 51b.

≪Tracking part≫
Tracking unit 51b acquires the object candidate image S _t output from the object candidate image generation unit 51a, tracking method of sequential learning type, in the present embodiment, based on the improved KCF (Kernelized Correlation Filter) method, tracking objects 4 that are included in the object candidate image S _t. The KCF method is a method of tracking an image of the object 4 itself while learning sequentially (as needed). For this reason, in the KCF method, even in a situation where occlusion frequently occurs in the object 4 due to sweeping by a sweeper, such as curling, the object 4 can be tracked robustly while learning how the latest object 4 looks. Also, in the KCF method, unlike a method using a general 2-class classifier such as Boosting, MIL (Multiple Instance Learning), SVM (Support Vector Machine), etc., the memory usage is suppressed by FFT (Fast Fourier Transform). However, the object 4 can be learned and tracked at high speed. Further, in the method using a general two-class classifier, a search window (subwindow) is randomly sampled from the vicinity of the predicted position of the object 4 and the discrimination process is performed. The object 4 can be tracked at high speed by analytically processing the image group with the search window shifted to FFT.

Tracking unit 51b using the KCF method, to obtain an output value f (x) by adding the bias b with applying a weight w to the input u (series of pixel values of the object candidate image S _t, the feature vector, etc.).
f (u) = <w, u> + b
= W ₁ u ₁ + w ₂ u ₂ +... + W _m u _m + b Formula (1)
The tracking unit 51b using the KCF method determines whether or not the object shown in the image is the target object 4 based on the obtained output value f (u).

  The tracking unit 51b calculates the optimum values of the weight w and the bias b by minimizing the objective function of Expression (2) in the learning phase. In Equation (2), L is a loss function, and L increases as the error between the estimation result f (u) and the true value v increases. The bias b is a regularization term added to prevent f (u) from becoming complicated due to the weight w taking a large value. λ is a hyperparameter that determines the balance of the regularization term (weight w). As the hyperparameter λ is increased, the regularization effect is increased (the influence of the weight w is increased).

In the present embodiment, the tracking unit 51b generates an image group in which the search range 5 is shifted by one pixel around the object 4 to be tracked, and the generated image group is represented by a circular matrix (circulant) represented by Expression (3). matirixes) Treat as C (u). In this cyclic matrix C (u), each row corresponds to an image shifted by one pixel. In Expression (4), F is Fourier transform, F ⁻¹ is Fourier inverse transform, and * is complex conjugate. u is the input and is an image group shifted by one pixel around the area of the object 4 to be tracked. v is the true value of u. In the cyclic matrix C (u), each row of the matrix corresponds to an image shifted by one pixel. That is, the right side of the equation (4) is obtained by performing inverse Fourier transform on the product (Hadamard product) of each element of the complex conjugate of the input u subjected to Fourier transform and the true value v subjected to Fourier transform.

  The tracking unit 51b can calculate the product of the circulant matrix C (u) and the true value vector v according to the equation (4) only in the Fourier space. Since the spatial domain convolution operation is equivalent to the frequency domain multiplication / division operation, the tracking unit 51b can realize high-speed processing that does not require repetitive processing by using the characteristics of the cyclic matrix. In the tracking process of the object 4, the tracking unit 51 b calculates f (u) using the weight w and the bias b calculated in the learning phase, and determines u where the f (u) is maximum as the position of the object 4 ( Coordinate).

The position of the object 4, that is, the two-dimensional coordinates (x, y) is represented by P _f ^C (i). Here, i represents the serial number of the position of the object 4, and C represents the number of the fixed cameras 10A and 10B. C takes three values: "1" indicating the first fixed camera 10A, "2" indicating the second fixed camera 10B, and "T" in the upper viewpoint coordinate system. f indicates a forward trajectory as viewed from the fixed camera. When a trajectory extending in the reverse direction by reversal is represented, r (reverse) is used as P _r ^C (i).

  The tracking unit 51b continues to track the object 4 until the object 4 stops. The tracking unit 51b can calculate the speed of the object 4 for each frame, and can determine the end of tracking based on the calculated speed of the object 4. The tracking unit 51b can end the tracking based on the operation result of the operation unit 30 by the operator. Further, when the object 4 is stationary in a state where it is hidden by a player or the like (when occlusion occurs), the tracking unit 51b acquires the stationary position of the object 4 based on the operation result of the operation unit 30 by the operator. be able to.

≪Search range setting part≫
The curling stone that is the object 4 moves while increasing in size from the back side to the near side of the screen. Here, since the existing KCF method is designed on the assumption that the size of the search range 5 is fixed, tracking tends to become unstable when tracking a curling stone. Therefore, the search range setting unit 51 c updates the size R (see FIG. 6) of the search range 5 based on the size r of the object 4.
R _{t + 1} = (r _t / r _t−1 ) · R _t (5)

Here, t is the frame number, r _t-1 is the length of one side (vertical side or horizontal side) of the object 4 in the previous frame, and r _t is one side (vertical side or horizontal side) of the object 4 in the current frame. Side), R _t is the length of one side of the search range 5 in the current frame, and R _{t + 1} is the length of one side of the search range 5 in the next frame.

That is, the search range setting unit 51c enlarges / reduces the search range 5 from the current frame to the next frame based on the enlargement / reduction ratio of the size of the object 4 (here, the dimension of one side) from the previous frame to the current frame. Determine the reduction ratio. Specifically, the search range setting unit 51c determines the size r _t-1 of the object 4 in the previous frame (see FIG. 6A), the size r _t of the object in the current frame (see FIG. 6B), and the current frame. The size R _t of the search range 5 (see FIG. 6B) is acquired from the object candidate image generation unit 51a and / or the tracking unit 51b, and based on the acquired sizes r _t−1 , r _t , R _t . Then, the size R _{t + 1} of the search range 5 in the next frame (see FIG. 6C) is calculated, and the search range 5 of the calculated size is set in the image of the next frame. That is, the tracking unit 51b tracks the object 4 using the search range 5 set (updated) by the search range setting unit 51c. The shape of the search range 5 may be a square having the same length of the vertical side and the horizontal side, or may be a rectangle that changes the length of the vertical side and the horizontal side according to the shape of the object 4.

  The search range setting unit 51c determines the position (center) of the search range 5 in the t + 1 frame based on the movement vector (speed) of the object between the t-1 frame and the t frame. Position) can be set.

  The search range setting unit 51c tracks the object 4 using the search range 5 having a predetermined size (for example, the initial size described above) when t is a predetermined number or less (for example, 3 to 5). Can do. Similarly, the search range setting unit 51c can set the search range 5 around the initial position of the designated object 4 when t is a predetermined number or less (eg, 3 or more and 5 or less). Thereby, the search range setting unit 51c can suitably set the search range 5 particularly at t = 1,2.

<Projection conversion unit (projection conversion device)>
The projective transformation unit 52 acquires an object position group P _f ^C that is a set of the positions P _f ^C (i) of the object 4, and uses the acquired object position group P _f ^C for CG drawing of the trajectory of the object 4. Projective transformation into the coordinate system. Here, since P _f ^C (i) is a two-dimensional coordinate system photographed by one fixed camera 10 (for example, 10A), it is photographed by another fixed camera 10 (for example, 10C) having a different angle of view. It is difficult to draw the locus of the object 4 at an accurate position with respect to the obtained image. Therefore, the projective transformation unit 52 uses the projective transformation to temporarily convert the position (coordinates) of the object 4 photographed by one fixed camera 10 to the upper viewpoint coordinate system, and then to another fixed camera 10. Convert to another corresponding coordinate system. Here, the upper viewpoint coordinate system is a coordinate system in a case where the curling sheet 2 is photographed from above the central portion. The projective conversion unit 52 includes a first conversion unit 52a and a second conversion unit 52b.

≪First conversion part≫
The first conversion unit 52a acquires the object position group P _f ^C output from the tracking unit 51b, performs projective conversion on the acquired object position group P _f ^C to the upper viewpoint coordinate system, and converts the conversion result into the second It outputs to the conversion part 52b. If the coordinates after conversion are (x _p , y _p ) and the coordinates before conversion are (x _c , y _c ),

_{x p = (h 1 x c} + h 2 y c + h 3) / (h 7 x c + h 8 y c +1)
_{y p = (h 4 x c} + h 5 y c + h 6) / (h 7 x c + h 8 y c +1)

It becomes.
Since this projective transformation is a mapping from a plane to a plane, the coordinates obtained after the transformation are also two-dimensional coordinates. Here, assuming a plane perpendicular to the Z-axis in the three-dimensional space and passing through the origin, Z = 0 is always obtained, so that a two-dimensional coordinate after projective transformation is obtained, so that one point on the three-dimensional coordinate is obtained. Can be specified. Here, since the curling stone as the object 4 moves in parallel on the ice, the object in the two-dimensional coordinate system photographed by the first fixed camera 10A is set by setting Z = 0 with the height direction as the Z axis. From the position of 4, the three-dimensional position of the object 4 can be calculated. In the projective transformation matrix H in such projective transformation, h ₁ ,..., H ₈ are unknown projective transformation parameters. The eight projective transformation parameters h ₁ ,..., H ₈ can be calculated based on four or more points of correspondence between images before and after conversion.

  The first conversion unit 52a causes the display unit 40 to display an image captured by the first fixed camera 10A. In addition, the first conversion unit 52a has four points (for example, for example) in the image captured by the first fixed camera 10A based on the operation result of the operation unit 30 by the operator who viewed the image displayed on the display unit 40. The coordinates before conversion are acquired at the four corners 2a to 2d) of the curling sheet 2, and the projective transformation matrix H is generated based on the acquired coordinates before conversion and the coordinates after conversion stored in advance.

≪Second conversion part≫
The second conversion unit 52b acquires the conversion result of the first conversion unit 52a, and uses the acquired conversion result (upper viewpoint coordinate system) as another fixed camera 10 (for example, the second fixed camera 10B or Projective transformation is performed on the coordinate system of the third fixed camera 10C).

≪Responding to changes in angle of view≫
By using the above-described projective transformation, the second conversion unit 52b makes the locus of the object 4 photographed by the first fixed camera 10A the correct position with respect to the image of the third fixed camera 10C having a different angle of view. To be able to draw. That is, the projective conversion unit 52 can display the trajectory of the object 4 taken at a different angle of view in the past game in accordance with the current angle of view of the fixed camera 10C.

First, the first conversion unit 52a, with respect to the first fixed camera 10A which is set in the first field angle, calculates a projective transformation matrix H _1, the object position group P _f ^C in the first field angle Obtained from the object tracking unit 51 (see FIG. 7A). Subsequently, the second conversion unit 52b calculates a projective transformation matrix H3 for the _third fixed camera 10C set to the second angle of view.

Subsequently, the first conversion unit 52a obtains the object position group P _f ^T in the upper viewpoint coordinate system by performing projective transformation on the object position group P _f ^C with H ₁ (see FIG. 7B). Subsequently, the second conversion unit 52b, by the inverse projective transformation object location group P _f ^T at H _3, a third object position group P _f of the fixed camera 10C, which is set to a second angle of view ^{C ′} is acquired (see FIG. 8). In this way, the projective transformation unit 52 can obtain the object position group P _f ^{C ′} suitable for drawing on the image having the second angle of view.

≪Inversion of locus≫
Curling is performed up to 10 ends, and the positions of the houses are alternately switched between odd ends (1, 3, 5, 7, 9) and even ends (2, 4, 6, 8, 10). Therefore, the stone throwing at the odd end (the trajectory of the object 4) and the stone throwing at the even end are in opposite directions, and the object position group by one first fixed camera 10A is simply combined with the image by the other second fixed camera 10B. Then, when trying to display the trajectories of the odd-numbered end and even-numbered end objects 4 at the same time, the trajectories of the objects 4 in the opposite direction are mixed.

  Here, the projective transformation unit 52 rotates the object position group of one of the odd end and the even end by 180 ° with respect to the point target, thereby setting the object position group of both the odd end and the even end as a locus in the same direction. .

First, the first conversion unit 52a, with respect to the fixed camera 10A installed on one side of the curling sheet 2, and calculates a projective transformation matrix H _1, to obtain the object position group P _f ^C from the object tracking unit 51 ( FIG. 7 (a)). Subsequently, the first conversion unit 52a obtains the object position group P _f ^T in the upper viewpoint coordinate system by performing projective transformation on the object position group P _f ^C with H ₁ (see FIG. 7B). Subsequently, the second conversion unit 52b rotates the object position group in the upper viewpoint coordinate system by 180 ° with respect to the point target by the equations (8) and (9) (see FIG. 9A).
P _r ^T (i, x) = W _A −P _f ^T (i, x) (8)
^{_{P r T (i, y)}} = H A -P f T (i, y) ... formula (9)
Here, P _f ^T (i, x) is the x coordinate before rotation in the upper viewpoint coordinate system, P _f ^T (i, y) is the y coordinate before rotation in the upper viewpoint coordinate system, and P _r ^T (i , X) is the x coordinate after rotation in the upper viewpoint coordinate system, and P _r ^T (i, y) is the y coordinate after rotation in the upper viewpoint coordinate system. Further, _{W A} is the width of the curling sheet 2 (constant), _{H A} is the curling sheet 2 height (constant).

Subsequently, the second conversion unit 52b, with respect to the fixed camera 10B disposed on the other side of the curling sheet 2, to calculate a projective transformation matrix H _2. Subsequently, the second conversion unit 52b subjects the object position group P _r ^T to the object position group P _r ^{C ′ of} the fixed camera 10B installed on the other side of the curling sheet 2 by performing the projective transformation with the H _2. ^' Is acquired (see FIG. 9B). In this way, the projective transformation unit 52 can obtain an object position group P _f ^{C ′} suitable for drawing on an image from the opposite side of the curling sheet 2.

  It should be noted that the object position group before the projective transformation and after the projective transformation (after the second transformation) in the projective transformation unit 52 is an area of the field (curling sheet 2) on the plane to be imaged that is perpendicular to the field. It is represented by the coordinate system image | photographed from the position (for example, diagonally upward) out of the area | region. Further, the object position group in the middle of the projective transformation (after one transformation) in the projective transformation unit 52 is taken from the upper and lower regions orthogonal to the field of the field (curling sheet 2) on the plane to be photographed. (A coordinate system for conversion. For example, an upper viewpoint coordinate system, a lower viewpoint coordinate system).

<Track generation unit (track generation device)>
The trajectory generation unit 53 generates a trajectory of the object 4 using the obtained object position group, and stores data related to the generated trajectory in the storage unit 54. Here, if the positions of the objects 4 in all the detected frames are connected, influences such as erroneous detection and detection omission occur, so that it is difficult to draw a smooth trajectory. Therefore, the trajectory generation unit 53 generates a trajectory of the object 4 so as to be drawn as a smooth curve by sampling several points of the object position group and performing spline interpolation on the sampled object positions.

Spline interpolation determines a local function (curve) using only a few local points instead of determining a function using all of the given points, and the determined curves are This is a method that is smoothly connected as a whole. Spline interpolation does not determine a single polynomial from all of the points, but approximates each interval with a relatively low-dimensional polynomial. The trajectory generation unit 53 uses cubic spline interpolation, and generates the trajectory k of the object 4 by the following equations (10) and (11). In Expressions (10) and (11), k is a function indicating a trajectory obtained by spline interpolation. j is a serial number of several local points (n points (x, y)) selected to be used for spline interpolation. In equations (10) and (11), the trajectory k is described separately for each x coordinate and y coordinate.

k _j (x) = p _j + q _j (x−x _j ) + r _j (x− _{j j} ) ² + s _j (x−x _j ) ³ ... (10)
k _j (y) = p _j + q _j (y−y _j ) + r _j (y−y _j ) ² + s _j (y−y _j ) ³ ... (11)
Where j = 0, 1,..., N−1.

The trajectory generation unit 53 passes through all of the points (x _j , y _j ), and the coefficients p _j , q _j , r _j , s _j match each point (x _j , y _j ). p _j , q _j , r _j , and s _j are set. Here, r ₀ = r _n−1 = 0 for both ends of the square coefficient r _j .

<Track output unit>
The trajectory output unit 55 reads out and reads out data related to the trajectory (a set of k _j (x) and k _j (y)) stored in the storage unit 54 based on the operation result of the operation unit 30 by the operator. Data on the locus is output to the inserter 60.

<Inserter>
The inserter 60 includes an image of the fixed camera 10 (for example, the second fixed camera 10B or the third fixed camera 10C) output from the switcher 20, and data (for example, the first track) output by the track output unit 55. The data related to the trajectory of the object 4 photographed by one fixed camera 10A is acquired, and based on the acquired data related to the trajectory, the trajectory is fixed to the fixed camera 10 (for example, the second fixed camera 10B or the third fixed camera 10A). The image is synthesized on the image of the camera 10C) and a synthesized image is output). In the example shown in FIG. 10, the object 4 included in the images captured by the first fixed camera 10 </ b> A and the second fixed camera 10 </ b> B during one game is compared with the image captured by the second fixed camera 10 </ b> B. All of the trajectories (for example, the trajectory by the first fixed camera 10A is a dotted line and the trajectory by the second fixed camera 10B is a solid line) are combined. The trajectory of the object 4 photographed by the first fixed camera 10A is actually from the house 3B side to the house 3A, but in such a composite image, the trajectory is converted from the house 3B side to the house 3A. Yes. For this reason, the trajectory of the object 4 photographed by the first fixed camera 10A and the trajectory of the object 4 photographed by the second fixed camera 10B are directed in the same direction. The curling game content can be suitably communicated to. An image in which the trajectory of the object 4 is combined is displayed on a display device (not shown).

<Operation example>
Next, an operation example of the object trajectory generation system 1 according to the embodiment of the present invention will be described in the order of tracking the object 4 and projective transformation of the trajectory (object position group) of the object 4. The flow of FIG. 12 and FIG. 13 is executed after the flow of FIG.

≪Tracking of objects≫
As shown in FIG. 11, the object candidate image generation unit 51a obtains the fixed camera image I _t taken by the fixed camera 10, and generates an object candidate image S _t from the acquired fixed camera image I _t (step S1). When the object candidate image _St is the first image (Yes in Step S2), the object candidate image generation unit 51a receives the initial designation of the object 4 (Yes in Step S3) and receives the initial setting of the object 4 to set the position of the initial search range 5 around the the first sheet of the object candidate image S _t (step S4).

When the object candidate image _St is the second or subsequent image (No in step S2) and after execution of step S4, the object candidate image generation unit 51a narrows down object candidates within the search range 5 (step S5). ). Subsequently, the tracking unit 51b, by using the KCF method for detecting an object 4 from the object candidate image _{S t} (step S6).

  Subsequently, the search range setting unit 51c updates the position and size of the search range 5 based on the detected position and size of the object 4 (step S7). Steps S1 to S7 are repeated until the object 4 stops (No in step S8).

  When the object 4 is stationary (Yes in step S8) and the occlusion occurs in the stationary object 4 (Yes in step S9), the tracking unit 51b receives a manual designation of the final position of the object 4. (Step S10). This flow ends when no occlusion has occurred in the stationary object 4 (No in step S9) and when the final position of the object 4 is manually designated (Yes in step S10).

≪Projective transformation of object trajectory: Response to changes in angle of view≫
As illustrated in FIG. 12, when the second conversion unit 52b receives manual designation of four points of the imaging result of the third fixed camera 10C (Yes in Step S21), the second conversion unit 52b is based on the coordinates of the four points designated manually. Te generates a projective transformation matrix _{H 3} (step S22). Subsequently, the first conversion unit 52a by projective transformation using projective transformation matrix H ₁ of the object position group P _f ^C is calculated in advance from the imaging result of the first fixed cameras 10A obtained in advance Then, the object position group P _f ^T in the upper viewpoint coordinate system is calculated (step S23). Subsequently, the second conversion unit 52b performs inverse projective transformation on the object position group P _f ^T using the projective transformation matrix H ₃ , whereby the object position group P _f ^{C ′} in the coordinate system of the third fixed camera 10C. Is calculated (step S24).

≪Projective transformation of object trajectory: Response to changes in angle of view≫
On the other hand, as illustrated in FIG. 13, when the second conversion unit 52b receives manual designation of four points of the photographing result of the second fixed camera 10B (Yes in step S41), the coordinates of the four points manually designated. based on, to generate a projective transformation matrix _{H 2} (step S42). Subsequently, the first conversion unit 52a by projective transformation using projective transformation matrix H ₁ of the object position group P _f ^C is calculated in advance from the imaging result of the first fixed cameras 10A obtained in advance Then, an object position group P _f ^T in the upper viewpoint coordinate system is calculated (step S43). Subsequently, the second conversion unit 52b calculates the object position group P _r ^T by rotating the object position group P _f ^T by 180 ° about the center of the curling sheet 2 (step S44). Subsequently, the second conversion unit 52b converts the object position group P _r ^T into the object position group P _r ^C in the coordinate system of the second fixed camera 10B by performing reverse projection conversion on the object position group P _r ^T using the projection conversion matrix H _2. Calculate (step S45).

  The object trajectory generation system 1 according to the embodiment of the present invention tracks the object 4 by the KCF method, which is a sequential learning type tracking method, and sets the search range 5 to be larger as the object 4 becomes larger. In addition, the object 4 that is likely to cause occlusion can be suitably tracked.

  Moreover, since the object locus generation system 1 tracks the object 4 using the object candidate image generated based on the color of the object 4, the object 4 can be tracked at a higher speed.

  Further, since the object locus generation system 1 performs projective transformation through the upper viewpoint coordinate system which is a coordinate system for conversion, the position group of the object 4 can be suitably converted to a coordinate system from a different viewpoint.

  In particular, the object trajectory generation system 1 converts the position group of the object 4 into the coordinate system of the fixed camera 10 from a different angle of view, or converts it into the coordinate system of the fixed camera 10 on the opposite side of the curling sheet 2. By doing so, the position group of the object 4 can be converted into a coordinate system suitable for combining with the image of another fixed camera 10.

  In addition, since the object locus generation system 1 generates the locus of the object 4 by spline interpolation, it is possible to prevent the influence of erroneous detection, detection omission, and the like and generate a smooth locus.

  In addition, since the object trajectory generation system 1 tracks the curling stone as the object 4, it can suitably track the curling stone which has a variable shooting size and is likely to be occluded. Further, the object trajectory generation system 1 can suitably display a curling stone trajectory captured at a different angle of view or from the opposite direction on a display device.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it can change suitably. For example, the sequential learning type tracking method in the tracking unit 51b is not limited to the improved KCF method. In addition, the position group of the object 4 stored in the storage unit 54 may be given the attribute of the position group based on the operation result of the operation unit 30 by the operator. Attributes include team (red, yellow), shot type (take shot, draw shot), player name, shot number, and the like. In this case, the trajectory output unit 55 can read out the position group of the object 4 having the attribute specified by the operation result from the storage unit 54 based on the operation result of the operation unit 30 by the operator, and output it to the inserter 60. . That is, the inserter 60 can display the trajectory of the object 4 having a specific attribute on a display device (not shown).

  Moreover, the coordinate system for conversion is not limited to the upper viewpoint coordinate system, and may be, for example, a lower viewpoint coordinate system in which the curling sheet 2 is virtually viewed from below the center.

  In addition, the object trajectory generation system 1, the object tracking unit (object tracking device) 51, the projection conversion unit (projection conversion device) 52, and the trajectory generation unit (trajectory generation device) 53 of the present invention are used for the object 4 other than the curling stone. Applicable to tracking and trajectory generation.

1 Object locus generation system 2 Curling seat (field)
3A, 3B House 4 Object (Stone)
5 Search Range 10, 10A, 10B, 10C Fixed Camera 51 Object Tracking Unit (Object Tracking Device)
51a Object candidate image generation unit 51b Tracking unit 51c Search range setting unit 52 Projection conversion unit (projection conversion device)
52a First conversion unit 52b Second conversion unit 53 Trajectory generation unit (trajectory generation device)

Claims (7)

An object tracking device for tracking an object included in a fixed camera image,
A tracking unit that tracks the object by a sequential learning type tracking method;
A search range setting that sets the search range in the frame of the fixed camera video so that the size of the search range of the object in the next frame increases as the size of the object included in the current frame of the fixed camera video increases. And
With
The tracking unit tracks the object within the search range set in the frame by the search range setting unit. An object candidate image generation unit that generates an object candidate image obtained by extracting object candidates included in the frame based on the color of the object;
The tracking unit tracks the object using the object candidate image,
The object tracking apparatus according to claim 1, wherein the search range setting unit sets the search range in the object candidate image. The object tracking device according to claim 1 or 2,
A projective transformation device for projective transformation of the coordinates of the object tracked by the tracking unit;
A trajectory for generating a trajectory of the object based on the position group of the object tracked by the object tracking device or the position group of the object tracked by the object tracking device and projectively transformed by the projective transformation device. A generating device;
With
The projective transformation device comprises:
A first conversion unit for projectively converting the coordinates of the object included in the fixed camera video to a coordinate system for conversion;
A second conversion unit for projectively converting the coordinates of the object converted into the coordinate system for conversion into a coordinate system from a different viewpoint from the fixed camera video;
An object trajectory generation system comprising: The first conversion unit converts the coordinates of the object taken from one side of the field into the coordinate system for conversion when the field is viewed from above,
The said 2nd conversion part converts the coordinate of the said object converted into the coordinate system for the said conversion into the coordinate system which looked at the said field from the one side at a different angle of view. The object trajectory generation system described. The first conversion unit converts the coordinates of the object taken from one side of the field into the coordinate system for conversion when the field is viewed from above,
The second conversion unit rotates the coordinates of the object converted into the coordinate system for conversion by 180 ° in the plane of the field, and the coordinates of the object rotated by 180 ° as the one side of the field. The object locus generation system according to claim 3, wherein the object locus generation system is converted to a coordinate system viewed from the other side opposite to. The object locus generation system according to any one of claims 3 to 5, wherein the locus generation device generates a locus of the object by spline interpolation. The object trajectory generation system according to any one of claims 3 to 6, wherein the object is a curling stone.

Images