JP2019066974A - Moving body tracking device, moving body tracking method, and moving body tracking program - Google Patents

Abstract

To track a target moving body with high accuracy using image data even when the moving body has few outstanding features in appearance and there are a plurality of images similar to that of the target moving body in frame images.SOLUTION: Even when a target moving body has few outstanding features in appearance and there are a plurality of images similar to that of the target moving body in frame images, the moving body is tracked with high accuracy by evaluating a predetermined feature (for example, a motion of a ball follows the law of motion and the ball is likely located above a table-tennis table) in a given environment (for example, table-tennis) from image data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a moving object tracking device, a moving object tracking method, and a moving object tracking program.

  Conventionally, an apparatus for tracking a moving body using an image is known. For example, Patent Document 1 describes an apparatus for detecting and tracking a moving object by evaluating the degree of color movement.

  Further, Patent Document 2 describes a device for detecting a moving object trajectory by detecting an amount of movement of pixels between frame images and superimposing an image indicating a position of a pixel at which the amount of movement is maximum. It is done.

International Publication No. 2012/127618 International Publication No. 2016/021143

  However, in the device described in Patent Document 1, since the moving body is detected and tracked by evaluating the degree of movement of the color, for example, color and shape such as table tennis ball and golf ball etc. It is difficult to track moving objects with few appearanceally distinctive features such as etc. without relying on those features.

  Also, for example, even if the moving object has an appearance feature such as color or shape, such as a volleyball or soccer ball, if a plurality of candidates are detected, one target should be specified. If you can not track down is difficult.

  In the device described in Patent Document 2, the trajectory of a moving body is detected from the movement amount of pixels between frame images to track the moving body. For example, in table tennis etc., in addition to the ball being used for competition, practice is also performed. There are many cases where there are multiple images similar to the ball for the purpose of tracking in multiple frame images, such as balls used on occasion and balls on the floor and spare balls possessed by referees, etc. There is also a possibility that an object having the following features is erroneously detected as a tracking target. In such a case, it is difficult to track moving objects with high accuracy.

  A moving object tracking device according to the present invention is a moving object tracking device that uses an image data to track the movement of an object having a particular feature in a given environment, and is detected from the image data in an image frame. A position information generation unit that generates position information of all objects having a specific feature; a candidate position holding unit that holds the position information of a plurality of candidates for the object that are likely to be tracking targets for each image frame; For the candidate of the first image frame for which the candidate position holding unit holds the position information, predicted position information in the second image frame continuous to the first image frame based on the motion model determined from the given environment Calculating the position information, calculating the position information generated by the position information generation unit from the image data in the second image frame, the position From the predicted position information in the second image frame, a likelihood score indicating the degree of coincidence of the position is calculated for each combination, and one or more of the second image frame is calculated based on the likelihood score. The candidate extraction unit which extracts the candidate and outputs position information of the extracted candidate to the candidate position holding unit, and the position information of the candidate extracted by the candidate extraction unit is determined from the given environment A target position identifying unit for identifying position information of the object which is the tracking target in the image frame based on an evaluation score indicating likelihood as a tracking target, and a position of the object identified by the target position identifying unit And a moving object tracking unit that tracks the movement of the object by tracing information in successive image frames.

  In the moving object tracking device according to the present invention, the candidate extraction unit may be configured to use the second image frame in which the candidates for the first image frame based on which the position prediction unit calculates the predicted position information are the same. An upper limit can be set on the number of object candidates.

  In the moving object tracking device according to the present invention, the candidate extraction unit can extract the candidates in at least one second image frame based on each of all the candidates of the first image frame.

  In the moving object tracking device according to the present invention, the candidate extraction unit can exclude the candidate in the second image frame whose likelihood score is not equal to or more than a threshold from the candidates.

  In the moving object tracking device according to the present invention, the position prediction unit can calculate predicted position information in the second image frame based on the position and velocity of the object in the first image frame as the motion model. .

  In the moving object tracking device according to the present invention, the target position specifying unit obtains the evaluation score from the position score based on the presence probability distribution of the target object and the motion score based on the velocity of the target object, and uses the tracking target in the image frame. The position information of a certain object can be specified.

  A moving object tracking method according to the present invention is a moving object tracking method for tracking the movement of an object having a specific feature in a given environment using image data, which is detected from the image data in an image frame. Position information generating step of generating position information of all objects having the specific feature; candidate position holding step of holding the position information of a plurality of candidates for the object likely to be tracking targets for each image frame And predicting the second image frame continuous with the first image frame based on a motion model determined from the given environment for the candidate of the first image frame for which the candidate position holding unit holds the position information. A position prediction step of calculating position information, and the position information generated by the position information generation unit from the image data in the second image frame A likelihood score indicating the degree of coincidence of positions is calculated for each combination from the predicted position information in the second image frame calculated by the position prediction unit, and the second image is calculated based on the likelihood score. A candidate extraction step of extracting one or more candidates in a frame and outputting position information of the extracted candidates to the candidate position storage unit; and position information of the candidates extracted by the candidate extraction unit; A target position specifying step of specifying position information of the object which is the tracking target in the image frame based on an evaluation score indicating the likelihood as the tracking target determined from the given environment; Moving object tracking step of tracing the movement of the object by tracing the position information of the object specified by , Characterized in that it comprises.

  The moving object tracking program according to the present invention is a moving object tracking program that causes a computer to use an image data to track the movement of an object having a particular feature in a given environment, in an image frame. A position information generation unit that generates position information of all objects having the specific feature detected from the image data; and holds position information of a plurality of candidates for the object likely to be tracking targets for each image frame A candidate position holding unit; a second image continuous with the first image frame based on a motion model determined from the given environment for the candidate of the first image frame for which the candidate position holding unit holds the position information; A position prediction unit that calculates predicted position information in a frame; and a position information generation unit that generates the position information generation unit from the image data in the second image frame. Based on the position information and the predicted position information in the second image frame calculated by the position prediction unit, a likelihood score indicating the degree of coincidence of the position is calculated for each combination, and the likelihood score is calculated based on the likelihood score. A candidate extraction unit that extracts one or more of the candidates in the second image frame and outputs position information of the extracted candidates to the candidate position storage unit; positions of the candidates extracted by the candidate extraction unit A target position identifying unit for specifying position information of the object which is the tracking target in the image frame based on an evaluation score indicating likelihood as the tracking target determined from the given environment from information; As a moving body tracking unit that tracks the movement of the object by tracing the position information of the object specified by the specifying unit in the successive image frames Characterized in that to ability.

  As described above, according to the present invention, it is possible to track a moving object with high accuracy while solving the above-mentioned problems.

FIG. 1 is a block diagram of an apparatus including a motion tracking device. It is a hardware block diagram of the apparatus shown in FIG. It is the figure which represented typically the image in the table tennis imaged with the camera. It is a flowchart which shows operation | movement of a moving body tracking device. It is the figure which showed typically the symbol used in demonstrating the operation | movement of a moving body tracking device. It is a flowchart which shows operation | movement of the candidate extraction part of a moving body tracking device. It is a flowchart which shows operation | movement of the target position identification part of a moving body tracking device. It is a flowchart shown about the candidate extraction process by the likelihood score which concerns on 2nd Embodiment. It is the figure which showed typically the candidate extraction process by the likelihood score which the candidate extraction part which concerns on 2nd Embodiment performs. It is a flowchart (the first half) shown about the candidate extraction process by the likelihood score which concerns on 3rd Embodiment. It is a flow chart (the second half) showing about candidate extraction processing by a likelihood score concerning a 3rd embodiment. It is the figure which showed typically the candidate extraction process by the likelihood score which the candidate extraction part which concerns on 3rd Embodiment performs.

  Embodiments of the present invention will be described with reference to the drawings. In addition, the overlapping description is abbreviate | omitted and the same code | symbol is attached | subjected to an identical or equivalent part in each drawing.

First Embodiment
The moving body tracking device according to the present embodiment is a device for tracking a moving body using an image. The moving object tracking device according to the present embodiment is, for example, an appearance such as color or shape that moves according to a motion law within a defined field, such as tracking the movement of a ball in ball games such as table tennis, golf, baseball, volleyball, etc. The present invention is preferably employed in an apparatus for tracking a moving object with few distinctive features with high accuracy. This is an example and does not limit the application to athletics only.

  FIG. 1 is a block diagram of an apparatus 2 provided with a moving object tracking apparatus 1. The device 2 shown in FIG. 1 is, for example, an electric appliance such as a smartphone, an information portable terminal, a digital camera, a game terminal, a television, etc., and has a hardware configuration shown in FIG. As shown in FIG. 2, the device 2 physically includes a central processing unit (CPU) 201, an input device 202, an output device 203, a main storage device (RAM / ROM) 204, an auxiliary storage device 205, and the like. Configured as a system. As the input device 202, a camera, a touch screen, a keyboard and the like, and as the output device 203, a display, a printer and the like are specifically configured. The device 2 may be configured to include a communication module or the like to connect to a network. Also, the moving object tracking device 1 may be configured as a computer system including a central processing unit (CPU) 201, an input device 202, an output device 203, a main storage device (RAM / ROM) 204, an auxiliary storage device 205, and the like.

  Each function of the moving object tracking device 1 and the device 2 is processed by reading predetermined computer software into the central processing unit (CPU) 201, the main storage unit (RAM / ROM) 204, etc. shown in FIG. This is realized by operating the input device 202 and the output device 203 under the control of) 201 and reading and writing data with the main storage device (RAM / ROM) 204 and the auxiliary storage device 205.

  As shown in FIG. 1, the device 2 includes a camera 21, a detection unit 22, a moving object tracking device 1, and a display unit 23. The moving object tracking device 1 further includes an information input unit 10, a position information generating unit 11, a candidate position holding unit 12, a position predicting unit 13, a candidate extracting unit 14, a target position specifying unit 15, and a moving object tracking unit 16.

  For example, an imaging device or the like is used as the camera 21 and has a function of capturing an image or a moving image. The camera 21 may be a single eye or a compound eye. The camera 21 has a function of outputting a plurality of captured images or moving images as an image frame to the detection unit.

  The detection unit 22 has a function of detecting all candidates that match the feature of the target object from the image data in the image frame input from the camera 21. For example, candidates are detected using pattern matching, a statistical learning method, or the like. The detection unit 22 has a function of outputting information on a candidate detected from image data in an image frame to the information input unit 10 of the moving object tracking device 1.

  The motion tracking device 1 tracks a target object by specifying and tracing position information in an image frame of a target object from information on a candidate detected from image data in an image frame input from the detection unit 22. It has a function. The moving object tracking device 1 has a function of outputting the position information of each frame of the specified target object to the display unit 23.

  The information input unit 10 has a function of outputting necessary information to the position information generation unit 11 from information on candidates detected from the image data in the image frame input from the detection unit 22. Necessary information includes at least information for specifying an image frame (hereinafter, image frame number) and information for specifying a position of a candidate (hereinafter, position information). The information input unit 10 has a function of outputting an image frame number to the position prediction unit 13. In addition, the information input unit 10 may have a function of extracting (hereinafter, frame dropping) image frames at a constant interval and outputting necessary information to the position information generation unit 11.

  The position information generation unit 11 has a function of generating position information of each candidate from information on candidates detected from image data in the image frame input from the information input unit 10. For example, if the camera 21 is a monocular, two-dimensional coordinate axes are set, and if the camera 21 is a compound eye, position information (coordinates) of each candidate is generated. The position information generation unit 11 has a function of reflecting the condition related to the position affecting the tracking of the moving body in the position information of the candidate. For example, it is a coordinate area of a table tennis table in tracking of a table tennis ball. The position information generation unit 11 has a function of outputting the generated position information to the candidate extraction unit 14.

  The candidate position holding unit 12 has a function of holding information on upper candidates inputted from the candidate extraction unit 14. The function of the candidate extraction unit 14 and the definition of the upper candidate will be described later. The candidate position holding unit 12 has a function of holding at least an image frame number, position information of a high candidate, and a likelihood score as information on the high candidate. The likelihood score will be described later.

  The position prediction unit 13 holds candidate position information on upper candidates in the image frame (first image frame) continuous with the image frame (second image frame) specified by the image frame number input from the information input unit 10 It has a function of inquiring to the unit 12 and acquiring it.

  Here, a continuous image frame is a state in which another image frame does not exist between two image frames, and the image frame numbers do not necessarily have to be continuous. For example, even if two image frames are continuous, there is a case where image frame numbers are not continuous because image frames originally existing between the frames are dropped. Further, the continuous image frame may be a continuous image frame whose time is before or a continuous image frame whose time is later. In the case of performing moving object tracking along the flow of time, the time is a previous continuous image frame, and in the case of performing moving object tracking retroactive to the flow of time, the time is a subsequent continuous image frame. The same applies to the successive image frames in the document pertaining to the present patent application.

  The position prediction unit 13 calculates predicted position information in the image frame (second image frame) for each upper candidate of consecutive image frames (first image frame) based on the information acquired from the candidate position holding unit 12. It has a function to calculate. The predicted position information is calculated, for example, from the position and velocity of the upper candidate. Further, for example, in the case of a table tennis ball, an exercise model is defined which takes into consideration information specific to the environment such as a change in speed by bouncing on the table tennis table. The position prediction unit 13 has a function of outputting the calculated predicted position information to the candidate extraction unit 14.

  The candidate extracting unit 14 detects the position information of the candidate detected from the image data in the image frame (second image frame) input from the position information generating unit 11 and the same image frame input from the position predicting unit 13 It has a function of calculating a likelihood score from predicted position information in two image frames). Here, the likelihood score is a score indicating the degree of coincidence of positions for each combination of position information and predicted position information for a candidate, and for example, the smaller the distance, the larger the score. The candidate extraction unit 14 has a function of inquiring the position prediction unit 13 of predicted position information necessary for calculation of the likelihood score. The candidate extraction unit 14 has a function of outputting the position information to the target position identification unit 15 with the number of candidates determined from the top of the likelihood score as the top candidates. In addition, the candidate extraction unit 14 has a function of outputting position information and the like of upper candidates to the candidate position holding unit 12.

  The target position specifying unit 15 has a function of calculating an evaluation score from the position information of the upper candidate input from the candidate extracting unit 14. Here, the evaluation score is a score indicating the likelihood as a tracking target. The evaluation score is defined by, for example, a combination of a position score and a motion score. Here, the position score is a score based on the presence probability distribution of the target object. For example, when the target object is a table tennis ball, the position score is high because the ball probability is high at the top of the table tennis table, and the position score is low because the ball probability is low at the bottom of the table tennis table. Here, the movement score is a score based on the velocity of the target object. For example, when the target object is a table tennis ball, the movement score is small when the magnitude of the velocity is below a certain level, and the movement score is high when the reflex movement is performed near the table tennis table. As a definition of the evaluation score, the position score and the motion score may be simply added, or may be defined by weighting and adding, multiplication and the like. The target position specifying unit 15 has a function of specifying the candidate position of the highest evaluation score as the position of the target object in the image frame. The target position specifying unit 15 has a function of outputting the specified position information of the target object to the moving object tracking unit 16.

  The moving body tracking unit 16 has a function of tracing and tracking the position information of the specified target object in each image frame input from the target position specifying unit 15. The moving body tracking unit 16 has a function of outputting the position information of the traced target object in each frame to the display unit 23.

  The display unit 23 has a function of displaying the locus of the target object by continuously displaying the position information of each frame of the target object input from the moving object tracking unit 16 of the moving object tracking device 1 on a display or the like, for example. Have.

  Next, the operation of the moving object tracking device 1 according to the present embodiment will be described. A moving object tracking device for tracking a ball in table tennis will be described as an example of the operation of the moving object tracking device according to the present invention in consideration of the ease of explanation and understanding.

  Describe the features to be considered when making table tennis a given environment. FIG. 3: is the figure which represented typically the image in the table tennis imaged with the camera 21. As shown in FIG. Since the ball has few distinctive features such as color and shape in appearance, when detecting the ball in the image, the detection accuracy of the detection unit 22 is anything other than the ball such as the face of the audience (304) Has the feature that it may be falsely detected. In addition to the ball (301) used in the competition, which is the target object, there is also a feature that other objects than the target object such as a spare ball (302) and a ball (303) rolling on the floor are detected. There is. For this reason, the target object is tracked excluding the candidate (304) other than the ball (301) used in the game and erroneously detected, and the candidate (302, 303) which is not the target object even if it is the ball. It is required to

  A feature to be considered when given table tennis as a given environment is, for example, a feature that it is necessary to cope with the change of the target object to another ball by changing the ball used by the player, etc. have.

  As a feature to be considered when using table tennis as a given environment, as shown in FIG. 3, the ball which is the target object has a feature of having a high probability of being present at the top of the table tennis table. In addition, the movement of the ball is characterized by the movement law, but it frequently stops turning (302) while performing a parabola movement and a movement (301) that changes direction (bounces) upon collision with a table tennis table. Also, it has a feature that straight movement (rolling) movement (303) at low speed is hard to occur. The accuracy is improved by considering these features in identifying the target object.

  FIG. 4 is a flowchart showing the operation of the moving object tracking device 1 according to the present embodiment. In addition, the candidate position holding part 12 of the moving body tracking device 1 is demonstrated as what hold | maintains the information about the high-order candidate in a 1st image frame before execution of FIG. 4 in consideration of the ease of description understanding.

  FIG. 5 is a view schematically showing symbols used to explain the operation of the moving object tracking device 1 according to the present embodiment. The t0 frame and the t frame are continuous image frames. Pn (t0) is the n-th high-level candidate in the t0 frame. Vn (t0) is the speed of Pn (t0). Qn (t) is the nth prediction candidate in t frame and is predicted as a position in t frame based on Pn (t0). Am (t) is the m-th of all candidates detected in t frame. Dmn (t) is the distance between Qn (t) and Am (t). The meanings of these symbols are the same in other documents relating to the present patent application.

  As shown in FIG. 4, first, the information input unit 10 inputs information on candidates detected from image data in an image frame (second image frame), and then necessary information from the input information (at least the image A process of extracting the frame number and the position information) and outputting it to the position information generation unit 11 and the position prediction unit 13 is performed (S401). At this time, frame dropping processing may be performed at the same time (S401). When the process of S401 ends, the process proceeds to position information generation process (S403) and position prediction process (S407).

  In the process of S403, the position information generation unit 11 converts the position information of the candidate input from the information input unit 10 into position information in a coordinate system suitable for the subsequent processes. In the example of tracking the ball in table tennis in the present embodiment, assuming that the camera 21 has a compound eye, the intersection point of the vertical line where the origin is lowered from the center of the table tennis table and the floor, the X axis is the long side direction of the table tennis table, Y The axis is the short side direction of the table tennis table, the Z axis is the vertical direction with respect to the table tennis table, and the upper side is positive and the lower side is negative. Moreover, according to the coordinate system, the area of the table tennis table used for the game is −1.370 ≦ X ≦ 1.370, −0.7625 ≦ Y ≦ 0.7625, Z = 0.76. When the process of S403 ends, the process moves to a candidate detection process (S409).

In the process of S407, the position prediction unit 13 detects the position of the upper candidate in the image frame (first image frame) continuous with the image frame (second image frame) specified by the image frame number input from the information input unit 10. A process of acquiring information on candidate position information (D405) is performed. Next, a process of calculating predicted position information in the second image frame for upper candidates of each first image frame is performed. Assuming that the first image frame is t0 frame and the second image frame is t frame, the predicted position Qn (t) can be obtained by the following equation in the example of tracking the ball in table tennis in this embodiment. Here, the unit system in the formula is the MKS unit system, and so forth.
Qn (t) = Pn (t0) + Vn (t0)
However, when all of the following conditions are satisfied, the ball is located above the table tennis table, and the predicted position Qn (t) can be obtained by the following equation in consideration of the bounce. Qn (t). X is the X coordinate of Qn (t), Qn (t). Y is the Y coordinate of Qn (t), Qn (t). Z represents the Z coordinate of Qn (t), and takes the same notation in the following.
Pn (t0). Z> 0.76, and −1.370 ≦ Qn (t). X ≦ 1.370, and −0.7625 ≦ Qn (t). In the case of Y ≦ 0.7625, Qn (t) = (Qn (t) .X, Qn (t) .Y, | Qn (t) .Z−0.76 | +0.76)
When the process of S407 ends, the process moves to a candidate extraction process (S409).

  In the process of S409, position information on the candidate detected from the image data in the second image frame input from the position information generation unit 11 by the candidate extraction unit 14 and the same second image input from the position prediction unit 13 The likelihood score is calculated from the predicted position information in the frame, and the upper candidate in the second image frame is extracted, and is output as the candidate position information (D405). Details of this process will be described later. When the process of S409 ends, the process moves to a target position identification process (S411).

  In the process of S411, the target position specifying unit 15 calculates an evaluation score from the position information of the upper candidate input from the candidate extraction unit 14, and sets the candidate position of the highest evaluation score as the position of the target object in the image frame. Perform processing to identify. Details of this process will be described later. When the process of S411 ends, the process proceeds to moving object tracking process (S413).

  In the process of S413, the moving body tracking unit 16 traces and tracks the position information of the specified target object in each image frame input from the target position specifying unit 15. When the process of S413 ends, the process proceeds to a display process (S415).

  In the process of S415, the display unit 23 continuously displays the position information of each frame of the target object input from the moving object tracking unit 16 to display the trajectory of the target object. When the process of S415 ends, the process proceeds to an end determination process (S417).

  In the process of S417, it is determined whether the process has been performed for all image frames input from the detection unit 22 to the information input unit 10. If it is determined in the process of S417 that the process has not been performed for all image frames, the process proceeds to S401 again. In the process of S401, after the information about the candidate detected from the image data in the next unprocessed image frame is input, the processes of S401 to S417 are executed. Thus, the processing from S401 to S417 is repeatedly performed until the processing is performed for all image frames. On the other hand, if it is determined in the process of S417 that the process has been performed for all image frames, the control process illustrated in FIG. 4 ends.

  Next, each process of subroutine shown in FIG. 4 will be described in detail. First, the details of the candidate extraction process (S409) will be described using FIG. FIG. 6 is a flowchart showing the operation of the candidate extraction unit 14 of the moving subject tracking device 1 according to the present embodiment.

  As described above, in the candidate extraction process (S 409), position information on the candidate detected from the image data in the second image frame input from the position information generation unit 11 by the candidate extraction unit 14 and the position prediction unit 13 The likelihood score is calculated from the prediction position information in the same input second image frame, and the top candidates in the second image frame are extracted. In the present embodiment, the number of upper candidates to be extracted will be described as a natural number N.

  As shown in FIG. 6, first, the candidate extraction unit 14 sets m = 1 and n = 1 as initial values (S601).

  In the process of S603, the candidate extraction unit 14 acquires the m-th Am (t) of all the candidates obtained in the process of S403.

  In the process of S605, the candidate extraction unit 14 acquires the n-th Qn (t) from the prediction candidate obtained in the process of S407.

In the process of S 607, the candidate extraction unit 14 calculates the likelihood score Lmn (t) in the same manner as the likelihood score Lmn (t) of the combination of Am (t) and Qn (t). )) And hold the result (D619). Lmn (t) is obtained from the distance Dmn (t) between Am (t) and Qn (t) by the following equation.
Lmn (t) = exp (−Dmn (t) ² )

  In the process of S609, the candidate extraction unit 14 determines whether Qn (t) is the last of the prediction candidates obtained in the process of S407. If it is determined that it is not the end of the prediction candidate, the process of adding 1 to n (S611) is performed, and then the process of S605 is performed again. Thus, the processing of S605 to S609 is repeatedly performed until the processing is performed for all the prediction candidates. On the other hand, if it is determined in the process of S609 that it is the last of the prediction candidates, the process proceeds to the process of S613.

  In the process of S613, the candidate extraction unit 14 determines whether Am (t) is the last of the candidates obtained in the process of S409. If it is determined that it is not the end of the candidate, the process of adding 1 to m (S615) is performed, and then the process of S603 is performed again. Thus, the processing from S603 to S613 is repeatedly performed until the processing is performed for all the candidates. On the other hand, in the process of S613, when it is determined that it is the last of the prediction candidate, the process proceeds to the candidate extraction process (S617) by the likelihood score.

  In the candidate extraction process (S617) based on the likelihood score, the candidate extraction unit 14 extracts the combination of the top N pairs of Am (t), Qn (t), and Lmn (t) as candidates by the likelihood score. The candidate extraction unit 14 ends the candidate extraction process and shifts to the main routine.

  Next, details of the target position identification process (S411) will be described using FIG. FIG. 7 is a flowchart showing the operation of the target position specifying unit 15 of the moving object tracking device 1 according to the present embodiment.

  In the process of S701, the target position specifying unit 15 sequentially acquires the candidates obtained in the process of S409.

  In the process of S703, the target position specifying unit 15 calculates a position score (Sp). In the example which tracks the ball in table tennis in this embodiment, a position score (Sp) is calculated | required by the following numerical formula in each case where a coordinate shows next.

-1. 370 ≦ X ≦ 1.370, and −0.7625 ≦ Y ≦ 0.7625, and Z ≧ 0.76, Sp = 1.0

-1. 370 ≦ X ≦ 1.370 and −0.7625 ≦ Y ≦ 0.7625, and Z ≦ 0.76 Sp = 0.0

Other than the above case, in the case of −0.7625 ≦ Y ≦ 0.7625, Sp = exp (− (| X | −1.370) ² )

In any case other than the above cases Sp = exp (-(2 (| Y | -0.7625) (| X | -1.370)) ² )

  In the process of S 705, the target position specifying unit 15 calculates an exercise score (Sm). In the example which tracks the ball in table tennis in this embodiment, a movement score (Sm) is calculated | required by the following numerical formula in each case where a coordinate shows next.

In the case of Vz = 0.0, Sm = 0.001
In the case other than the above, Sm = min {1.0, 1- | V / 30-1 |}

  In the process of S707, the target position specifying unit 15 calculates an evaluation score from the position score and the motion score. In the example of tracking the ball in table tennis in this embodiment, the evaluation score is calculated by adding the position score and the motion score.

  In the process of S709, the target position specifying unit 15 determines whether the evaluation score calculated in the process of S707 is the largest as compared with the evaluation scores so far. If it is determined that the target position is the largest, the processing proceeds to step S711, and the target position D 715 is replaced with Am (t) determined to have the largest evaluation score. If it is determined not to be the maximum, the process proceeds to the process of S713.

  In the process of S713, the target position specifying unit 15 determines whether the candidate is the last. If it is determined that the process is not the last, the process proceeds to the process of S701 again. Thus, the processing of S701 to S713 is repeatedly performed until the processing is performed for all the candidates. On the other hand, when it is determined that it is the last, Am (t) determined to have the highest evaluation score is specified as the target position D 715, and the target position identification process is ended and the routine is shifted to the main routine.

  Next, a moving object tracking program for causing a computer system to function as the moving object tracking device 1 will be described. The configuration of the computer system is as shown in FIG.

  The motion tracking program comprises a main module, an input module and an arithmetic processing module. The main module is a part that controls image processing in an integrated manner. The input module operates the computer system to obtain image data in the image frame. The arithmetic processing module includes a position information generation module, a candidate position holding module, a position prediction module, a candidate extraction module, a target position specifying module, and a moving body tracking module. The functions realized by executing the main module, the input module and the arithmetic processing module are the information input unit 10 of the moving object tracking device 1, the position information generation unit 11, the candidate position holding unit 12, the position prediction unit 13, the candidate extraction unit The functions of the target position specifying unit 15 and the moving object tracking unit 16 are the same as those in FIG.

  The motion tracking program is provided by, for example, a storage medium such as a ROM or a semiconductor memory. Also, a moving body tracking program may be provided via a network.

  As described above, in the moving object tracking device 1 according to the present embodiment, the position information generation unit 11 generates information about candidates detected from image data in an image frame (second image frame) input from the information input unit 10. The position information of the candidate is generated, the predicted position in the image frame (second image frame) is calculated by the position prediction unit 13, and the candidate extraction unit 14 acquires the predicted position and the candidate position holding unit 12. A predetermined number of upper candidates in the second image frame are calculated from the upper candidates in the first image frame, and are output to the candidate position holding unit 12 and held. In this manner, by holding the plurality of upper candidates determined for each image frame in the candidate position holding unit 12, it is possible to use a spare ball or a floor other than the ball used for the competition as the target object. It is possible to determine an appropriate target object in an environment having a feature that other than the target object such as a rolling ball is detected.

  Further, in the moving object tracking device 1 according to the present embodiment, the target position specifying unit 15 specifies the position information of the object which is the tracking target in the image frame based on the evaluation score, and the evaluation score is the position score and the motion score Defined by a combination of The movement tracking unit 16 tracks the movement of the object by tracing the position information of the specified object in the successive image frames. By calculating the evaluation score by combining the position score and the exercise score in a given environment (for example, table tennis) in this manner, a plurality of candidates are detected because there are few appearanceally distinctive features such as color and shape. In such a case, it is possible to specify the target object with high accuracy.

  In the description of the operation of the moving object tracking device 1 according to the present embodiment, the moving object tracking device for tracking a ball in table tennis is described as an example of the operation of the moving object tracking device according to the present invention. However, in comparison with Example 1 as Example 1, an example of ball tracking in volleyball (Example 2), an example of ball tracking in baseball pitching (Example 3), an example of ball tracking in golf Example 4 will be described.

  The configuration shown in FIG. 1 of the moving object tracking device 1 according to the present embodiment is the same as that of the first embodiment and the second to fourth embodiments, and the second embodiment is also similar in that it is realized as the device 2 shown in FIG. The contents of the processing are different because the conditions according to a given environment are different. Specifically, the following points are different.

  The coordinate system suitable for the process of converting the position information input from the input information unit 10 by the position information generation unit 11 is different.

  The conditions regarding the position which affects tracking of the moving body which the positional infomation production | generation part 11 reflects positional information in production | generation differ.

  The formulas for calculating the predicted position by the position prediction unit 13 are different.

  The mathematical formulas for calculating the position score by the target position specifying unit 15 are different.

  The mathematical formulas for calculating the exercise score by the target position specifying unit 15 are different.

The differences are summarized in the table for Example 1 to Example 4. Example 1 is shown in Table 1, Example 2 in Table 2, Example 3 in Table 3, and Example 4 in Table 4.

Second Embodiment
The moving object tracking device according to the second embodiment has the same configuration as that of the moving object detection device 1 according to the first embodiment shown in FIG. 1 and is also the same as the device 2 shown in FIG. In the functions, the functions of the moving object detection device 1 according to the first embodiment and the candidate extraction unit 14 are different. Therefore, the candidate extraction unit 14 will be mainly described, and the overlapping description with the first embodiment will be omitted.

  The candidate extraction unit 14 according to the second embodiment receives position information from the position prediction unit 13 for the candidates detected from the image data in the image frame (second image frame) input from the position information generation unit 11. It has a function of calculating the likelihood score from predicted position information in the same image frame (second image frame). The candidate extraction unit 14 has a function of inquiring the position prediction unit 13 of predicted position information necessary for calculation of the likelihood score. The candidate extraction unit 14 has a function of outputting the position information to the target position identification unit 15 with the number of candidates determined from the top of the likelihood score as the top candidates. In addition, the candidate extraction unit 14 has a function of outputting position information and the like of upper candidates to the candidate position holding unit 12. The point which has these functions is the same as that of candidate extraction part 14 concerning a 1st embodiment.

  The candidate extraction unit 14 according to the second embodiment predicts that the candidate in the image frame (first image frame) based on which the position prediction unit 13 calculates the predicted position information is the same when extracting the high-order candidate It has a function of setting an upper limit on the number of candidates based on position information. This point is different from the candidate extraction unit 14 according to the first embodiment.

  The aforementioned difference is due to the difference in the candidate extraction process (S617) according to the likelihood score shown in FIG. FIG. 8 is a flowchart showing a candidate extraction process (S617) based on the likelihood score according to the second embodiment.

  As shown in FIG. 8, first, the candidate extracting unit 14 sets n = 1 as an initial value (S801).

  In the process of S803, the candidate extraction unit 14 acquires a combination of Am (t), Qn (t), and Lmn (t) as data. The candidate extracting unit 14 sorts the data in the descending order of the likelihood score Lmn (t), and arranges data on which Qn (t) is the basis of the calculation in the descending order of Lmn (t).

  In the process of S805, the candidate extraction unit 14 leaves from the top to S natural numbers in the likelihood score as candidates based on Qn (t).

  In the process of S807, the candidate extraction unit 14 determines whether Qn (t) is the last of the prediction candidates obtained in the process of S407. If it is determined that it is not the end of the prediction candidate, the process of adding 1 to n (S809) is performed, and then the process of S803 is performed again. Thus, the processing of S803 to S807 is repeatedly performed until the processing is performed for all the prediction candidates. On the other hand, if it is determined in the process of S807 that the end of the prediction candidate is determined, the process proceeds to S811.

  In the process of S811, the candidate extraction unit 14 extracts a combination of Am (t), Qn (t), and Lmn (t) having high likelihood scores as top candidates from the candidates remaining in the process of S805. After that, the candidate extraction process based on the likelihood score is ended, and the process proceeds to the main routine.

  FIG. 9 is a diagram schematically illustrating candidate extraction processing (S617) based on the likelihood score performed by the candidate extraction unit 14 according to the second embodiment. In FIG. 9, data are arranged in descending order of likelihood score Lmn (t) from left to right in each column for each prediction candidate Qn (t). In the process of S805, the candidate extraction unit 14 leaves S up to S natural numbers from the top in the likelihood score as a candidate based on Qn (t), which is an example in the case of S = 4 in FIG. , Indicates that shaded data is excluded from candidates. In the process of S811, the candidate extraction unit 14 extracts the top N candidates with the highest likelihood score from the candidates remaining in the process of S805, but in FIG. The degree score corresponds to extracting top N top candidates.

  The candidate extraction unit 14 according to the second embodiment, as described in FIG. 9, extracts a top candidate from candidates equally left for each of the prediction candidates Qn (t), the specific prediction candidate Qn (t). Less bias towards. As a result, top candidates remain in addition to the candidates existing in the vicinity of the leading candidate. Therefore, for example, when the target object for a different ball is changed due to a change in the ball used by the player, etc., or when an object other than the ball detected temporarily and erroneously is tracked, etc. It is possible to return to the desired target object and continue tracking.

Third Embodiment
The moving object tracking device according to the third embodiment has the same configuration as that of the moving object detection device 1 according to the first embodiment shown in FIG. 1, and is similar to the point realized as the device 2 shown in FIG. In the functions, the functions of the moving object detection device 1 according to the first embodiment and the candidate extraction unit 14 are different. Therefore, the candidate extraction unit 14 will be mainly described, and the overlapping description with the first embodiment will be omitted.

  The candidate extraction unit 14 according to the third embodiment receives position information from the position prediction unit 13 with respect to the candidates detected from the image data in the image frame (second image frame) input from the position information generation unit 11. It has a function of calculating the likelihood score from predicted position information in the same image frame (second image frame). The candidate extraction unit 14 has a function of inquiring the position prediction unit 13 of predicted position information necessary for calculation of the likelihood score. The candidate extraction unit 14 has a function of outputting the position information to the target position identification unit 15 with the number of candidates determined from the top of the likelihood score as the top candidates. In addition, the candidate extraction unit 14 has a function of outputting position information and the like of upper candidates to the candidate position holding unit 12. The point which has these functions is the same as that of candidate extraction part 14 concerning a 1st embodiment.

  The candidate extraction unit 14 according to the third embodiment, when extracting high-order candidates, likelihoods for all the candidates of the image frame (first image frame) on which the position prediction unit 13 bases calculation of predicted position information. It has a function of extracting candidates in at least one second image frame based on the condition that the score is equal to or higher than the threshold. This point is different from the candidate extraction unit 14 according to the first embodiment.

  The aforementioned difference is due to the difference in the candidate extraction process (S617) according to the likelihood score shown in FIG. 10A and 10B are flowcharts showing the candidate extraction process (S617) based on the likelihood score according to the third embodiment.

  As shown in FIG. 10A, first, the candidate extraction unit 14 sets n = 1 as an initial value (S1001).

  In the process of S1003, the candidate extraction unit 14 acquires a combination of Am (t), Qn (t), and Lmn (t) as data. The candidate extracting unit 14 sorts the data in the descending order of the likelihood score Lmn (t), and arranges data on which Qn (t) is the basis of the calculation in the descending order of Lmn (t).

  In the process of S1005, the candidate extraction unit 14 outputs a data array (D1007) of a combination of Am (t), Qn (t) and Lmn (t) generated in the process of S1003.

  In the process of S1009, the candidate extraction unit 14 determines whether Qn (t) is the last of the prediction candidates obtained in the process of S407. If it is determined that it is not the end of the prediction candidate, the process of adding 1 to n (S1011) is performed, and then the process of S1003 is performed again. Thus, the processing of S1003 to S1009 is repeatedly performed until the processing is performed for all the prediction candidates. On the other hand, if it is determined in the process of S1009 that it is the end of the prediction candidate, the process proceeds to the process of S1013 shown in FIG. 10B.

  As shown in FIG. 10B, the candidate extraction unit 14 sets n = 1 and j = 1 as initial values (S1013).

  In the process of S1015, the candidate extracting unit 14 determines the j-th Am (t), Qn (t), Lmn (t) in descending order of the likelihood score Lmn (t) for the data candidate (D1007) to the prediction candidate Qn (t). Get the data of the combination of).

  In the process of S1017, the candidate extraction unit 14 determines whether Lmn (t) is equal to or more than a threshold value for the data acquired in the process of S1015. If it is determined that the threshold value is exceeded, the combination of Am (t), Qn (t) and Lmn (t) acquired in the process of S1015 is extracted as a top candidate (S1019), and then the process proceeds to the process of S1021. . On the other hand, when it is determined that the threshold value is not exceeded, the processing shifts to the processing of s1021.

  In the process of S1021, the candidate extraction unit 14 determines whether the number of upper candidates extracted so far is smaller than N or not. If it is determined that the number of upper candidates is smaller than N, the process proceeds to s1023. On the other hand, when it is determined that the number of upper candidates is not smaller than N, the candidate extraction process based on the likelihood score is ended, and the process proceeds to the main routine.

  In the process of S1023, the candidate extraction unit 14 determines whether Qn (t) is the last of the prediction candidates obtained in the process of S407. If it is determined that it is not the end of the prediction candidate, the process of adding 1 to n (S1025) is performed, and then the process of S1017 is performed again. Thus, the processes of S1017 to S1023 are repeatedly performed until the process is performed for all the prediction candidates. On the other hand, if it is determined in the process of S1013 that the end of the prediction candidate is reached, the process proceeds to S1027.

  In the process of S1027, the candidate extraction unit 14 determines whether j is the tail of the prediction candidate Qn (t) of the data array (D1007). If it is determined that j is not the tail of the prediction candidate Qn (t), the process of adding 1 to j (S1029) is performed, and then the process proceeds to S1017 again. Thus, the process of S1017 to S1027 is repeatedly executed until j reaches the end of the data array for the prediction candidate Qn (t). On the other hand, if it is determined in the process of S1027 that j is the tail of the prediction candidate Qn (t), the candidate extraction process using the likelihood score is ended, and the process proceeds to the main routine.

  FIG. 11 is a diagram schematically illustrating candidate extraction processing (S617) based on the likelihood score performed by the candidate extraction unit 14 according to the third embodiment. In FIG. 11, data are arranged in descending order of likelihood score Lmn (t) from left to right in each column for each prediction candidate Qn (t). The data array (D1007) generated by the processing of S1001 to S1011 shown in FIG. 10A is schematically shown in FIG. Whether the likelihood score is equal to or higher than the threshold value in the process of S1017 while moving sequentially from the first row for each column of FIG. 11 in the process of S1023 downward in the process of S1027 while moving sequentially from the first column to the left for each row in FIG. Make a decision on In FIG. 11, the non-shaded portion corresponds to the likelihood score equal to or higher than the threshold.

  The candidate extraction unit 14 according to the third embodiment, as described in FIG. 11, extracts a top candidate from candidates evenly left for each of the prediction candidates Qn (t), a specific prediction candidate Qn (t). Less bias towards. Moreover, since the candidate whose likelihood score is smaller than the threshold is not the top candidate, it is superior to the second embodiment in that the candidate with low possibility (for example, Q3 (t) in FIG. 11) can be excluded. . As a result, top candidates remain in addition to the candidates existing in the vicinity of the leading candidate. Therefore, for example, when the target object for a different ball is changed due to a change in the ball used by the player, etc., or when an object other than the ball detected temporarily and erroneously is tracked, etc. It is possible to more accurately return to the target object and continue tracking.

DESCRIPTION OF SYMBOLS 1 moving body tracking device 10 information input part 11 position information production | generation part 12 candidate position holding part 13 position prediction part 14 candidate extraction part 15 target position identification part 16 moving body detection part 2 apparatus provided with moving body tracking apparatus 21 camera 22 detection part 23 display part 201 Central Processing Unit (CPU)
202 Input device 203 Output device 204 Main storage device (RAM / ROM)
205 Auxiliary storage device 301 Target object (ball used in competition)
302 Non-target object (spare ball)
303 A non-target object (ball rolling on the floor)
304 Object that is not the target (the face of the audience)

Claims (8)

A moving object tracking device that uses an image data to track the movement of an object having a particular feature in a given environment, comprising:
A position information generation unit that generates position information of all objects having the specific feature detected from the image data in an image frame;
A candidate position holding unit that holds the position information of a plurality of candidates for the object likely to be tracking targets for each image frame;
For the candidate of the first image frame for which the candidate position holding unit holds the position information, the predicted position in the second image frame continuous with the first image frame based on the motion model determined from the given environment A position prediction unit that calculates information;
About each combination from the position information generated by the position information generation unit from the image data in the second image frame and the predicted position information in the second image frame calculated by the position prediction unit A likelihood score indicating the degree of coincidence of positions is calculated, one or more of the candidates in the second image frame are extracted based on the likelihood score, and position information of the extracted candidates is used as the candidate position. A candidate extraction unit to output to the holding unit;
Position information of the object as the tracking target in the image frame based on an evaluation score indicating likelihood as the tracking target determined from the given environment from the position information of the candidate extracted by the candidate extraction unit A target position identification unit for specifying
A moving body tracking unit that tracks the movement of the object by tracing the position information of the object specified by the target position specifying unit in successive image frames;
A moving body tracking device comprising:   The candidate extraction unit sets an upper limit on the number of candidates of the object in the second image frame in which the candidate of the first image frame based on which the position prediction unit is based on the calculation of the predicted position information is the same The moving body tracking device according to claim 1.   The moving object tracking device according to claim 1, wherein the candidate extraction unit extracts the candidates in at least one second image frame based on each of all the candidates of the first image frame.   The moving object tracking device according to claim 3, wherein the candidate extraction unit excludes the candidate in the second image frame in which the likelihood score is not equal to or more than a threshold from the candidate.   The moving object tracking according to any one of claims 1 to 4, wherein the position prediction unit calculates predicted position information in the second image frame based on the position and velocity of the object in the first image frame as the motion model. apparatus.   The target position specifying unit specifies the position information of the object which is the tracking target in the image frame by obtaining the evaluation score from a position score based on the presence probability distribution of the target object and a motion score based on the velocity of the target object. The moving body tracking device according to claim 5. A motion tracking method for tracking motion of an object having a specific feature in a given environment using image data, comprising:
Generating position information of all objects having the specific feature detected from the image data in an image frame;
A candidate position holding step of holding the position information of a plurality of candidates for the object likely to be tracking targets for each image frame;
For the candidate of the first image frame for which the candidate position holding unit holds the position information, the predicted position in the second image frame continuous with the first image frame based on the motion model determined from the given environment Position prediction step for calculating information;
About each combination from the position information generated by the position information generation unit from the image data in the second image frame and the predicted position information in the second image frame calculated by the position prediction unit A likelihood score indicating the degree of coincidence of positions is calculated, one or more of the candidates in the second image frame are extracted based on the likelihood score, and position information of the extracted candidates is used as the candidate position. A candidate extraction step to be output to the holding unit;
Position information of the object as the tracking target in the image frame based on an evaluation score indicating likelihood as the tracking target determined from the given environment from the position information of the candidate extracted by the candidate extraction unit Target position identification step for identifying
A moving body tracking step of tracking the movement of the object by tracing the position information of the object specified by the target position specifying unit in successive image frames;
A moving object tracking method comprising: A motion tracking program that causes a computer to use an image data to track the movement of an object having a particular feature in a given environment,
A position information generation unit that generates position information of all objects having the specific feature detected from the image data in an image frame;
A candidate position holding unit that holds the position information of a plurality of candidates for the object likely to be tracking targets for each image frame;
For the candidate of the first image frame for which the candidate position holding unit holds the position information, the predicted position in the second image frame continuous with the first image frame based on the motion model determined from the given environment Position prediction unit that calculates information,
About each combination from the position information generated by the position information generation unit from the image data in the second image frame and the predicted position information in the second image frame calculated by the position prediction unit A likelihood score indicating the degree of coincidence of positions is calculated, one or more of the candidates in the second image frame are extracted based on the likelihood score, and position information of the extracted candidates is used as the candidate position. Candidate extraction unit to output to the holding unit,
Position information of the object as the tracking target in the image frame based on an evaluation score indicating likelihood as the tracking target determined from the given environment from the position information of the candidate extracted by the candidate extraction unit Target position identification unit for identifying
A moving body tracking unit that tracks the movement of the object by tracing the position information of the object specified by the target position specifying unit in successive image frames;
A moving object tracking program characterized in that it functions as: