JP4695615B2 - Video object trajectory image synthesis apparatus and program thereof - Google Patents

Description

  The present invention relates to a trajectory image synthesizing apparatus for a video object that synthesizes a trajectory image indicating a trajectory of a subject with a video obtained by photographing a moving subject, and a program thereof.

  Conventionally, in a sports program such as golf, if a subject is captured while being telephoto by a camera, the overall movement of the subject may not be easily transmitted to the viewer. For example, in a golf broadcast program, in the first shot of a short hole, an image of a camera installed near the green is generally broadcast, but the golf ball is small and can hardly be confirmed in a wide-angle image, so while chasing the ball with telephoto Taken. Although this telephoto image shows the partial movement of the ball, it is difficult to grasp the overall ball trajectory such as what course the ball flew to and the height of the ball.

Therefore, in recent sports broadcast programs, various virtual images are produced based on camera posture information such as camera pan, tilt and zoom. For example, in golf relay, the distance between the ball and the cup is displayed on the green, or the distance line is drawn with virtual CG (computer graphics) at the point of the first hit, making it difficult to convey only with the camera image. The device is designed to make the sense of distance easy to understand.
For example, Patent Document 1 discloses a method for calculating the three-dimensional position coordinates of a flying ball based on the principle of triangulation based on the positions of two cameras, the optical axis direction, and the angle of view value.

Furthermore, in Patent Document 2, using radio wave radars installed at a plurality of locations in a golf course, the radio wave is applied to the golf ball, and the flight speed and distance of the ball are obtained from the Doppler frequency and phase difference of the reflected wave, A technique for calculating the three-dimensional position coordinates of a ball by the principle of triangulation is disclosed.
Also, a device that calculates the position coordinates of the ball by the triangulation method from the positions of the two cameras, the optical axis direction and the angle of view, and measures the three-dimensional position of the flying ball for each frame (Non-Patent Document 1) Etc. exist.
Japanese Patent Laying-Open No. 2003-42716 (paragraphs 0032 to 0034, FIG. 3) JP-A-8-266701 (paragraphs 0039 to 0040, FIG. 3) Yuko Yamauchi, Daiichiro Kato, Hideo Noguchi, Kazuo Abe, Kazuo Sunami, “Flying object trajectory display system“ shot view ”—Visualization of the three-dimensional position of a golf ball”, Information Processing Society of Japan Research Report, Vol. 1995, No. .63, pp31-36

  It is desirable to operate the golf relay within the range of existing photography equipment. In other words, it must be operated without adding new equipment and photographers. In general, since golf relay is operated with one virtual camera per hole, the device of Patent Document 1 that requires two cameras is difficult to use.

  In addition, it is desirable that one camera can continuously extract from the sky until it stops, but in the methods of Patent Literature 1 and Patent Literature 2 and the device of Non-Patent Literature 1, the background and the ball area are dropped during the fall of the ball. There is a problem that the ball is easy to be erroneously extracted due to a change in the luminance difference between the two and the foreground area near the ground immediately before landing, and a large number of foreground areas close to the image characteristics of the ball. Furthermore, immediately after the shot, the ball speed changes suddenly from the stationary state to the initial speed that is faster than the speed at which the ball reaches the sky, making it difficult to extract the ball. There is a problem that it is easy to lose sight of the ball due to difficulty in predicting changes in speed and moving direction due to bouncing.

  For example, in the apparatus described in Non-Patent Document 1, three points of a hitting point, a falling point, and a stationary point that are difficult to extract due to a complicated background near the ground are manually designated. A blank section where no subject is extracted is generated between each of these points and the extracted point in the sky, but a smooth trajectory is displayed by smooth interpolation using a spline curve or the like. However, it has been desired to display a more accurate trajectory by continuously extracting from the sky to falling and stationary.

  Therefore, in the present invention, in view of such a problem, a moving trajectory of a small video object that can be operated by one camera and difficult to grasp the overall movement is continuously extracted, It is an object of the present invention to provide a video object trajectory image synthesis device for synthesizing an image superimposed on a camera video and a program therefor.

  The present invention has been made to achieve the above object, and the video object trajectory image synthesizing device according to claim 1 is a video obtained by photographing a moving subject with a camera, and the subject when the subject is photographed. A moving trajectory of the video object corresponding to the subject is calculated from camera parameters including information on the posture and angle of view of the camera, a trajectory image representing the calculated moving trajectory of the video object, and a video shot by the camera A trajectory image synthesizing device for a video object for synthesizing a search object, a search position update setting means, a threshold value determination means, an object region selection means, a two-dimensional predicted position calculation means, an object three-dimensional position calculation means, and an object 3 A dimension position storage unit, a trajectory image generation unit, and a trajectory image synthesis unit are provided.

  According to this configuration, the trajectory image composition device for a video object sets the search range of the video object in the video frame constituting the video by the search position update setting means. The trajectory image synthesis device generates a luminance histogram that is a luminance histogram of a video frame by threshold determination means, and extracts a video object from the search range set by the search position update setting means based on the luminance histogram. A threshold value for luminance and whether to extract a region with higher or lower luminance than the threshold value are determined, and an image of the determined region is extracted as a foreground image including a video object. The trajectory image synthesis device selects the video object by using the image feature of the video object from the foreground image extracted by the threshold value determination unit by the object region selection unit. The trajectory image composition device indicates the position of the video object in the video frame next to the video frame from the position in the video frame of the video object selected by the object area selection unit by the two-dimensional predicted position calculation unit. A two-dimensional predicted position is calculated. The trajectory image synthesizing device uses the object three-dimensional position calculation means to determine the object at the time when the video frame was shot based on the position in the video frame of the video object selected by the object area selection means and the camera parameters. A three-dimensional position indicating a position in the real space is calculated. The trajectory image synthesizing apparatus stores the three-dimensional position obtained by the object three-dimensional position calculation means by the object three-dimensional position storage means. The trajectory image synthesizing apparatus generates a trajectory image based on the three-dimensional position stored in the object three-dimensional position storage unit by the trajectory image generation unit. The trajectory image synthesis device can synthesize and output the trajectory image generated by the trajectory image generation means and the video frame by the trajectory image synthesis means, and can further output the two-dimensional predicted position by the search position update setting means. The search range of the next video frame is set based on the two-dimensional predicted position calculated by the calculation means.

  Thereby, the trajectory image composition device extracts a video object based on the luminance histogram from a video obtained by photographing a moving subject with a camera and a camera parameter including information on the posture state of the camera, and moves the video object. A trajectory can be calculated, and a trajectory image representing the calculated movement trajectory of the video object can be combined with the video shot by the camera.

  The video object trajectory image composition device according to claim 2 is the video object trajectory image composition device according to claim 1, wherein the position of the video object selected by the object region selection means in the video frame is 3D predicted position calculating means for sequentially predicting a 3D predicted position indicating a position of the subject corresponding to the video object in the next video frame in real space from the camera parameter, and calculating the 3D predicted position Screen coordinate conversion means for projecting and transforming the three-dimensional predicted position obtained by the means to calculate the position of the video object corresponding to the subject in the next video frame, and obtained by the screen coordinate conversion means. A sequence of points indicating the position in the video frame Trajectory curve interpolation means for interpolating using the object area selection means, and the object region selection means includes the video object candidates included in the foreground image of the next video frame from the curve generated by the trajectory curve interpolation means. The video object is selected based on the distance to each area.

  As a result, the trajectory image composition device obtains the three-dimensional predicted position by screen coordinate conversion using the three-dimensional predicted position sequentially predicted from the position of the video object in the video frame and the camera parameter. The video object is selected using the curve generated by interpolating the sequence of the determined points with a curve.

  The video object trajectory image composition apparatus according to claim 3 is the video object trajectory image composition apparatus according to claim 2, wherein the subject is based on the three-dimensional predicted position obtained by the three-dimensional predicted position calculation means. Subject height determining means for determining whether or not the height in real space is lower than a preset height, and the search position update setting means is less than the subject height determining means. The search range wider than the search range of the video frame preceding the video frame is set in the video frame next to the video frame at the determined time point.

  As a result, the trajectory image synthesis apparatus, based on the three-dimensional predicted position, in the video frame next to the video frame at the time when the height of the subject in the real space is determined to be lower by the subject height determination unit, A search range wider than the previous search range is set.

  The video object trajectory image composition apparatus according to claim 4 is the video object trajectory image composition apparatus according to any one of claims 1 to 3, wherein the video image storage means for storing the video image and the camera parameter are stored. Camera parameter storage means, and the search position update setting means inputs the time-series video frames constituting the video stored in the video storage means in reverse order, and the video frames in the video frames A search range is set, and the object three-dimensional position calculation means inputs the time-series camera parameters stored in the camera parameter storage means in reverse order to calculate the three-dimensional position.

  Accordingly, the video frames in which the subject is photographed are input in the reverse order of the time series constituting the video to set the search range of the video object, and the camera parameters are stored in the reverse order of the stored time series. The three-dimensional position can be calculated.

  The video object trajectory image synthesis program according to claim 5, wherein the moving object image is captured by a camera, and the camera parameter includes information on the posture and angle of view of the camera when the object is captured. A three-dimensional position indicating the position of the subject in the real space is calculated and stored in a storage device connected to the outside, and a trajectory image representing a movement locus of the video object corresponding to the subject is calculated based on the three-dimensional position. In order to synthesize the generated trajectory image and the video imaged by the camera, the computer is searched, update position setting means, threshold value determining means, object area selecting means, two-dimensional predicted position calculating means, and object three-dimensional position calculating. And function as a means, a trajectory image generation means, and a trajectory image synthesis means.

  According to this configuration, the trajectory image composition program of the video object sets the search range of the video object in the video frame constituting the video by the search position update setting unit. The trajectory image synthesis program generates a luminance histogram that is a luminance histogram of the video frame by threshold determination means, and based on the luminance histogram, the video object from the search range set by the search position update setting means A threshold value for extracting the luminance and whether to extract a region with higher or lower luminance than the threshold value, and foreground image including the video object as an image of the determined region Extract as The trajectory image synthesis program selects a video object by using an image feature of the video object from the foreground image extracted by the threshold value determination unit by an object region selection unit. The trajectory image synthesis program indicates the position of the video object in the next video frame from the position in the video frame of the video object selected by the object area selection unit by the two-dimensional predicted position calculation unit. A two-dimensional predicted position is calculated. The locus image synthesizing program is a method in which the object three-dimensional position calculating unit captures the video frame based on the position of the video object selected by the object region selecting unit in the video frame and the camera parameter. The three-dimensional position indicating the position of the subject in real space is calculated and stored in the storage device. The trajectory image synthesis program generates the trajectory image based on the three-dimensional position stored in the storage device by a trajectory image generation means. The trajectory image synthesis program synthesizes and outputs the trajectory image generated by the trajectory image generation means and the video frame by a trajectory image synthesis means.

  Accordingly, the trajectory image composition program extracts a video object based on the luminance histogram from a video obtained by photographing a moving subject with a camera and a camera parameter including information on the posture state of the camera, and moves the video object. A trajectory can be calculated, and a trajectory image representing the calculated movement trajectory of the video object can be combined with the video shot by the camera.

  According to the first aspect of the present invention, since the video captured by one camera and the camera parameters obtained from the camera are used, it is possible to operate with one camera and the luminance is a histogram of the luminance of the video frame. Based on the histogram, the threshold value for extracting the subject (video object) and whether to extract a region higher than this threshold value or a lower region are determined. However, the subject can be extracted continuously even if the brightness other than the subject increases or decreases. Therefore, a trajectory image can be generated from the extracted result and superimposed on the camera video.

  According to the second aspect of the present invention, the three-dimensional predicted position in the real space of the subject is calculated from the video signal of the camera that has photographed the subject and the camera parameters including information related to the posture state of the camera, and By generating a trajectory curve indicating a movement trajectory, it is possible to prevent an object extraction error due to movement in a complicated terrain shape where bounce cannot be predicted.

  According to the third aspect of the present invention, the landing of the subject is automatically determined by determining that the landing has been made on the ground when the height falls below a preset height from the three-dimensional predicted position in the real space of the subject. By expanding the search range, it is possible to generate a trajectory curve by extracting with high accuracy up to a stationary point even after the subject has landed on the ground.

  According to the fourth aspect of the present invention, an image obtained by photographing a moving subject by a camera and a camera parameter including information on the posture state of the camera are used as a starting point from a video frame when the subject approaches the sky. Reading from the storage device in the reverse order in time series, tracing the subject from the time when the subject (video object) is easily detected in the video frame to the time when the subject that is difficult to detect is fired. Thus, the trajectory up to the time when the subject is fired (during tee shot) can be extracted stably and with high accuracy.

  According to the fifth aspect of the present invention, since the video captured by one camera and the camera parameters obtained from the camera are used, it is possible to operate with one camera and the luminance is a histogram of the luminance of the video frame. Based on the histogram, the threshold value for extracting the subject (video object) and whether to extract a region higher than this threshold value or a lower region are determined. However, the subject can be extracted continuously even if the brightness other than the subject increases or decreases. Therefore, a trajectory image can be generated from the extracted result and superimposed on the camera video.

  Next, the best mode for carrying out the present invention will be described with reference to the drawings as appropriate. Here, a case where a moving subject is a golf ball and a trajectory image of a golf ball (video object) photographed by one camera is combined with a video image will be described as an example.

[Configuration of Trajectory Image Synthesizer (First Embodiment)]
First, the configuration of a video object trajectory image synthesis apparatus according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing an overall configuration of a video object trajectory image synthesis apparatus according to the first embodiment of the present invention.
As shown in FIG. 1, a trajectory image composition device 1 for a video object includes a subject movement start detection unit 2, a subject rise determination unit 3, a search position update setting unit 4, a threshold determination unit 5, and an object region selection unit. 6, prediction curve calculation means 7, landing determination means 8, two-dimensional prediction position calculation means 9, object three-dimensional position calculation means 10, video storage means 11, camera parameter storage means 12, subject image feature A storage unit 13, an object three-dimensional position storage unit 14, a trajectory image generation unit 15, and a trajectory image synthesis unit 16 are provided.

  The trajectory image composition device 1 of a video object receives a camera image obtained by photographing a subject that is launched and moved by the camera, and camera parameters including information on the posture and angle of view of the camera when the subject is photographed. The video object in the video frame constituting the video is extracted, the three-dimensional position indicating the position of the subject in the real space is obtained, the trajectory image of the video object in the video frame is generated, the trajectory image of the video object and the camera video Are synthesized.

  The subject movement start detection means 2 detects the point in time when the subject (golf ball) is launched into the sky, a sudden change in posture information due to the camera swing-up operation, that is, the camera between the previous video frame and the current video frame. Is detected based on the difference in camera parameters (tilt angle) related to the posture information. Here, since the subject is a golf ball, the subject movement start detection means 2 detects the timing (at the time of shot) when the golf ball is hit.

  When the subject movement start detection means 2 detects the time of a shot, the subject movement start detection means 2 notifies the subject rise determination means 3 of the time (frame number or time) at which it was detected that the golf ball was hit. The video and the camera parameters when the subject is photographed are output to the subject rise determination unit 3, and the camera video and the camera parameters are stored in the video storage unit 11 and the camera parameter storage unit 12, respectively. Let

Here, the subject movement start detection means 2 detects the tilt angle θ _t of the camera parameter corresponding to the video frame at time t (frame number) and the time one unit time before every unit time (one cycle of the video frame). When the difference between the tilt angle θ _t −1 of the video frame at (t−1) becomes larger than a predetermined value Ta as shown in the following formula (1), It is detected as the timing when it was hit.

  The subject rise determination unit 3 determines whether or not the subject has entered the sky based on the camera image input from the subject movement start detection unit 2 or the camera parameters when the subject is photographed. The point in time (frame number or time t) is detected. The subject rise determination unit 3 outputs the detected time t, the camera image, and camera parameters when the subject is photographed to the search position update setting unit 4, and the time information is a camera parameter storage unit. 12 is stored.

  Note that the time t when the subject has approached the sky is when the video object at the time when the subject is first launched into the sky (at the time of the shot) is extracted by the trajectory image synthesis device 1B (see FIG. 2). It is also used as the time to start reading video frames and camera parameters.

  Since the object rise determination means 3 has many video objects resembling the ball near the ground immediately after the shot and the camera operation is not stable, it tends to cause a ball extraction error. It is determined whether or not it has risen to start extraction. The timing for starting the extraction of the ball is determined when the tilt angle of the camera becomes larger than a predetermined value T3 as shown in the following equation (2).

  Here, the subject rise determination unit 3 determines whether or not the ball has approached the sky based on the tilt angle of the camera. As another method, the subject rise determination unit 3 is based on a luminance histogram that is a luminance histogram of the video signal in the video frame. Thus, it is possible to determine whether the background video is near the ground or has already approached the sky.

  The search position update setting unit 4 sets the search range of the ball (video object) in the video frame of the camera video input from the subject rise determination unit 3 or read from the video storage unit 11. The search position update setting means 4 first sets the search position from the time (frame number) when the subject (video object for) read (output) from the camera parameter storage means 12 approaches the sky. Then, the search position update setting unit 4 starts setting the search range when the video signal in the video frame at the time t when it is detected from the subject ascent determination unit 3 that the ball has approached the sky is input. In order to start extraction, initial conditions for the position and width of the ball search range are specified. The initial condition is set to a wider search range at the center of the screen in consideration that the ball is generally shot at the center of the video frame screen.

  Further, the search position update setting unit 4 relates to the setting of the search range of the video object in the video frame after the next time t + 1, and the next determination unit 8 determines that the subject has not landed. For the video frame at the time, the search range of the video object is set based on the two-dimensional position input from the two-dimensional predicted position calculation means 9, or the video frame next to the video frame determined that the subject has landed is set. For the video frame at the time, the video object search range is set larger than the search range of the previous video frame. The information (image coordinates, area) of the search range set here and the video signal photographed by the camera are output to the threshold value determination means 5.

  Up to this point, the camera parameter storage means 12 stores the time (frame number) at which the subject is launched into the sky and the time (frame number) at which the subject has approached the sky. Note that the search position update setting unit 4 goes back in time sequence in the reverse order to extract the video objects in the video frame from the time when the ball approaches the sky to the time when the ball is launched. And the camera parameters can be read out from the video storage unit 11 and the camera parameter storage unit 12, respectively, and the search range can be sequentially updated and set.

  The threshold value determination means 5 generates a luminance histogram that is a histogram of the luminance of the video frame in the video signal, and based on the luminance histogram, at least a threshold value for the sky with respect to the luminance information of the video signal of the video object (formula (3 described later) ) And a threshold value for the vicinity of the ground (T2 in equation (3) to be described later) and a foreground image and a background image are separated. As shown in FIG. 3A, the threshold value determination unit 5 includes a luminance histogram generation unit 51, a luminance threshold setting unit 52, and a foreground / background separation unit 53.

  The luminance histogram generation unit 51 sequentially generates a luminance histogram, which is a histogram of luminance information, for each video frame from the video signal in the video frame input from the search position update setting unit 4. The luminance histogram generated here is output to the luminance threshold setting means 52. Generally, in the sky, the luminance of the video object region is low with respect to the high-luminance background video, and on the ground, the luminance of the video object region is high with respect to the low-luminance background video. Thus, the brightness of the background video and the video object area (ball area) is reversed while the subject is rising.

  The brightness threshold setting means 52 utilizes the fact that the relative brightness between the background video and the ball area is reversed while the ball is rising, and the brightness of the background video of the video object generated by the brightness histogram generation means 51 is used. By sequentially referring to the histogram, it is determined whether the ball area is near the ground or approaching the sky, and the luminance threshold for extracting the ball area is switched. Further, the luminance threshold value setting means 52 determines the luminance range of the area extracted by the foreground / background separation means 53 described later from the threshold value when the ball area is determined to be sky based on the determination result of the luminance histogram. On the contrary, when the ball area is determined to be near the ground, it is set to a range higher than the threshold value.

  Strictly speaking, the luminance threshold value determination unit 52 sets the luminance range indicating the ball area depending on whether the frequency of the luminance histogram indicating high luminance is larger than the frequency of the luminance histogram indicating low luminance. Therefore, the brightness threshold value determination means 52 does not determine whether the ball area is actually approaching the sky or near the ground. For example, when the background of a camera image includes a mountain, a building, or the like, even if the ball area is approaching the sky, the background brightness is assumed to be lower than the brightness threshold value indicating the ball area. . However, even in this case, the brightness threshold value determination means 52 pays attention to the brightness histogram of the video frame regardless of whether the ball area is actually approaching the sky or near the ground. Therefore, the apparatus 1 can extract the ball area continuously as described below.

  For example, when the ball moves from the ground to the sky, the luminance threshold setting unit 52 gradually becomes dominant in the high-luminance background. Therefore, the frequency of the luminance histogram indicating the high-luminance background indicates the low-luminance background. When the frequency of the luminance histogram is exceeded, the luminance range of the ball area is set to 0 to T1. Conversely, when the ball moves from the sky to the ground, the low brightness background gradually becomes dominant, so the frequency of the brightness histogram indicating the low brightness background exceeds the frequency of the brightness histogram indicating the high brightness background. At this point, the luminance range of the ball area is set to T2 to 255. This process is shown in Equation (3). Here, the luminance takes a value of 0 to 255, the boundary luminance value between the high luminance background and the low luminance background is T3, and the frequency at luminance i is hist (i).

Here, the luminance threshold setting means 52 sequentially refers to the luminance histogram for each video frame, compares the frequency of the high luminance background and the low luminance background, and determines the threshold switching timing. The time when one frequency exceeds a certain threshold value (threshold value for the frequency in the luminance histogram) may be set as the switching timing.
The luminance threshold setting means 52 determines the luminance range of the ball area by an upper limit value T1 of the low area and a lower limit value T2 of the high area. These values are used as image features of the ball area extracted in the past. The upper limit value T1 and the lower limit value T2 of the luminance range can be dynamically determined by taking out the image area of the video object that is the target graphic from the subject image feature storage unit 13 and referring to it. Since the area in the current frame can be estimated from the ball area extracted in the past frame, the P tile method is effective as a threshold value determination method.

  The luminance threshold value setting means 52 uses the P tile method, as shown in the following formula (4), first, the area of the entire image is S, the area of the target graphic is S0, and the ratio p of the area of the target graphic to the entire image is set. Ask.

  Also, the brightness threshold setting means 52 counts the number of pixels from the larger luminance value when the target graphic is brighter than the background and from the smaller luminance value when the target graphic is darker than the background. The brightness value when the value is as shown in (5) is set as a threshold value. Here, the number of counted pixels is N0, and the total number of pixels is N.

  The foreground / background separation unit 53 extracts a region corresponding to the luminance range of the ball region set by the luminance threshold setting unit 52 as a foreground region (foreground image). The image information of the foreground area extracted here is output to the object area selection means 6. Thereby, the foreground / background separation means 53 extracts a region having a lower luminance than the threshold as the foreground region when the background is sky, that is, high luminance, and has a luminance higher than the threshold when the background is near the ground, that is, when the luminance is low. Extract the region as a foreground region. Returning to FIG.

  The object area selecting means 6 uses the object image characteristics for the foreground area extracted by the foreground / background separating means 53, with respect to the image characteristics such as the size and shape of the video object created in the past video frame in advance. The ball area is selected by reading from the storage means 13 and referring to it. That is, when there are a plurality of ball area candidates (when there are a plurality of foreground areas), the object area selection unit 6 selects the ball area closest to the target image feature. Since the image feature of the ball changes for each video frame by the camera operation, the average of the image features in the past several frames is output as the target image feature. Further, here, the object region selection unit 6 preferentially selects a region close to the prediction curve input from the trajectory curve interpolation unit 73 of the prediction curve calculation unit 7 as a video object (see FIG. See 3 for details).

  The coordinates and area indicating the position of the ball area in the video frame in the video signal selected by the object area selecting means 6 from the object area selecting means 6 to the prediction curve calculating means 7 and the two-dimensional predicted position calculating means 9. To the object three-dimensional position calculation means 10. Further, the object area selecting unit 6 outputs an average of the image features in the past several frames to the subject image feature storing unit 13.

  The prediction curve calculation means 7 predicts the three-dimensional position in the real space based on the coordinates (two-dimensional prediction position) and the area indicating the position of the video object in the video frame obtained by the object region selection means 6. The position of the video object up to the predetermined number of video frames is predicted by using the dimensional position prediction, and the prediction curve of the video object obtained by interpolating these positions is calculated. As shown in (b), a three-dimensional predicted position calculation means 71, a screen coordinate conversion means 72, and a trajectory curve interpolation means 73 are provided.

  The three-dimensional predicted position calculating means 71 predicts the corresponding three-dimensional predicted position in the next video frame of the ball that is the video object, based on the video object extracted by the object area selecting means 6. Here, the three-dimensional predicted position calculation means 71 sequentially estimates the three-dimensional predicted position of the ball for each video frame using an extended Kalman filter.

As shown in the following equation (6), the three-dimensional predicted position calculation means 71 converts the state quantity X into a position (x, y, z) and velocity (v _x , v _y , v) in a three-dimensional orthogonal coordinate space. _z), and acceleration _{(a x,} a _y, and _{a z),} the image coordinates indicating the center of gravity of the video objects extracted observables y from the camera image _{(X g, Y g),} and defined as the area (a). Superscript: ^T represents transposition of the matrix.

Further, as shown in the following equation (7), F is a state transition matrix with one frame as a unit time, and a motion model is obtained by adding a system noise v _t to a parabolic motion under a constant gravity g. It was. Note that Q is the covariance matrix of the system noise v _t . Also, I and O are 3 × 3 unit matrix and zero matrix, and t is a frame number (time). For example, E [v _t ] indicates an expected value of the system noise v _t .

  Then, the three-dimensional predicted position calculation means 71 sequentially predicts the position of the ball according to the motion model according to the state estimation formula shown in the following formula (8) every unit time. Here, P is an error covariance matrix of state quantities.

Here, as shown in the following equation (9), an observation model Y obtained by adding an observation noise w _t having a covariance matrix R to a function h (X _t ) for projecting a three-dimensional position onto a screen. _t . Note that the initial position coordinates (x, y, z) of the ball are determined in advance by performing calibration before photographing. The initial velocity (v _x , v _y , v _z ) of the ball and the area formula coefficient s are determined from a reference shot trajectory before imaging. Further, the pan angle of the camera parameters is φ, the tilt angle is θ, and the focal length is f.

  When the observed amount is obtained, the three-dimensional predicted position calculating unit 71 updates the state quantity X and the elements of the error covariance matrix P of the state quantity according to the update formula shown in the following formula (10), and after the update Let X ′ be the state quantity, and P ′ be the error covariance matrix of the updated state quantity. H is an observation sensitivity matrix.

As described above, the three-dimensional predicted position calculation means 71 updates the observation when observation is obtained based on the state estimation expression shown in Expression (8) and the update expression shown in Expression (10). While estimating the state according to the motion model, the three-dimensional predicted position of the ball is predicted sequentially.
Further, the three-dimensional predicted position of the ball (subject) obtained by the three-dimensional predicted position calculating means 71 is output to the landing determining means 8.

  The screen coordinate conversion means 72 projects and converts the three-dimensional predicted position for each frame of the ball (subject) obtained by the three-dimensional predicted position calculation means 71 to a position (image coordinate) in the next video frame. is there. The screen coordinate conversion unit 72 projects the three-dimensional predicted position of the ball for each frame calculated by the three-dimensional predicted position calculation unit 71 to the screen coordinates corresponding to the position in the video frame, and is arranged in the current video frame. A prediction point sequence including a plurality of prediction point sequences is obtained, and the image coordinates of the prediction point sequence are output to the trajectory curve interpolation unit 73.

  The trajectory curve interpolation unit 73 obtains a prediction curve by interpolating the prediction point sequence on the current screen coordinates of the video object (ball) obtained by the screen coordinate conversion unit 72 with, for example, a quadratic curve. It is. Then, the trajectory curve interpolation unit 73 outputs the image coordinates of the prediction curve to the object region selection unit 6.

  The landing determining means 8 has landed the ball (subject) on the ground based on the three-dimensional predicted position of the video object calculated by the three-dimensional predicted position calculating means 71 for the video object in the video frame in the video signal. As shown in FIG. 3C, the apparatus includes a subject height determination unit 81, a landing time determination unit 82, and a coordinate conversion unit 83.

The subject height determination unit 81 determines whether the altitude of the subject is below the reference value based on the three-dimensional predicted position of the subject (ball) obtained by the prediction curve calculation unit 7. That is, the subject height determination unit 81 determines the height element (z) from the position of the ball (position coordinates (x, y, z) of the state quantity x) sequentially predicted by the three-dimensional predicted position calculation unit 71 for each video frame. _t), as shown in equation (11) below, at the stage where falls below the height reference value H _G defined in advance as the ground level, it is determined that the ball has landed, the position of the landing point And output to the coordinate conversion means 83.

The landing time determination means 82 receives from the subject height determination means 81 the result of determining whether or not the ball has landed. If the result is that the ball has landed, the frame number of the video frame or the time t The information of _g is output to the search position update setting means 4.

  The coordinate conversion unit 83 converts the three-dimensional predicted position of the landing point of the ball determined by the subject height determination unit 81 into image coordinates in the video frame. Then, the coordinate conversion unit 83 outputs the image coordinates in the video frame indicating the landing point to the search position update setting unit 4.

Based on the coordinates and area indicating the position of the video object selected by the object region selection means 6, the two-dimensional predicted position calculation means 9 determines the two-dimensional predicted position (image coordinates) of the video object in the next video frame. This is predicted and output to the search position update setting means 4. The two-dimensional predicted position calculation means 9 calculates the two-dimensional predicted position by using the state estimation formula shown by the formula (8) and the update formula shown by the formula (10). Using the function h (x _t ) that is projected onto the screen with respect to the position coordinates expressed in the Cartesian coordinates, that is, the expression (9), the three-dimensional position is converted into the two-dimensional position (image coordinates), A two-dimensional predicted position can be calculated.

  The two-dimensional predicted position calculation means 9 calculates the two-dimensional predicted position of the video object in the next video frame based on the coordinates and area indicating the position of the video object selected by the object region selection means 6. At this time, as described above, the calculation is not limited to the method using the extended Kalman filter, and the calculation can also be performed using linear prediction.

The two-dimensional predicted position calculation means 9 predicts the two-dimensional predicted position of the ball, which is a video object, by linear prediction, and the video obtained from the video frames at the previous two times (t−1 and t−2). Using the image coordinates P _t−1 and P _t−2 of the object, the image coordinates P _t = [x, y] ^T of the video object of the video frame at time t are _expressed by the following equation (12): It can also be predicted.

  Based on the coordinates and area indicating the position of the video object for each video frame selected by the object area selection unit 6 and the camera parameters read from the camera parameter storage unit 12, the object three-dimensional position calculation unit 10 The three-dimensional position coordinates of the ball, which is a video object, are calculated, and the three-dimensional position coordinates are stored in the object three-dimensional position storage means 14.

  The video storage means 11 stores the video signals of the video frames taken by the camera obtained by the subject movement start detection means 2 in time series, and is constituted by a general storage means such as a hard disk. Is done. The video signal of the video frame stored here is read out in the forward or reverse direction by the search position update setting unit 4 and the trajectory image synthesis unit 16 with respect to time series.

  The camera parameter storage unit 12 stores the camera parameters obtained by the subject movement start detection unit 2 and related to the posture state of the video frame taken by the camera in time series, and is output from the subject movement start detection unit 2. The time (frame number) when the captured subject is launched into the sky and the time (frame number) when the subject output from the subject ascent determination means 3 reaches the sky are stored. Consists of means. The camera parameters stored here are read out in the forward or reverse direction by the search position update setting unit 4 and the trajectory image generation unit 15 with respect to time series.

  The subject image feature storage unit 13 is obtained by averaging the image features of the video objects in the past several frames obtained by the object region selection unit 6 (target image features: size, shape, luminance, etc. ), The upper limit value T1 of the luminance of the low luminance background, the lower limit value T2 of the luminance of the high luminance background, and the boundary luminance value T3 of the high luminance background and the low luminance background are stored as luminance threshold information. It is constituted by general storage means such as a hard disk. The image features of the video object stored here are read out in the forward direction or the reverse direction by the threshold value determination means 5 and the object area selection means 6 with respect to time series.

  The object three-dimensional position storage means (storage device) 14 stores the three-dimensional position coordinates of the video object in the video frame calculated by the object three-dimensional position calculation means 10. It is constituted by a storage means. The three-dimensional position coordinates of the video object stored in the video frame stored here are read in the forward direction or the reverse direction by the trajectory image generation means 15 with respect to time series.

  The trajectory image generation means 15 reads out the three-dimensional position coordinates of the video object in the video frame stored in the object three-dimensional position storage means 14 and the camera parameters stored in the camera parameter storage means 12. A trajectory image is generated for each video frame, and the trajectory image is output to the trajectory image synthesis means 16.

  The trajectory image synthesis means 16 inputs the trajectory image for each video frame generated by the trajectory image generation means 15 and the video frame of the video signal stored in the video storage means 11 and combines them. By doing so, a video signal on which the trajectory image of the video object (ball) is superimposed is generated. Here, the video on which the trajectory image is superimposed is output to the outside.

  As a result, the trajectory image synthesis device 1 starts from extracting a video object in a video frame at the time when a subject that has been difficult to hit on the ground until the subject reaches the stationary point after reaching the sky. The movement trajectory of the section can be extracted continuously.

  The trajectory image synthesis apparatus 1 can realize each unit as a function program in a computer, and can also operate the trajectory image synthesis program by combining the function programs.

[Operation of Trajectory Image Synthesizer (First Embodiment)]
Next, the operation of the trajectory image synthesis apparatus 1 will be described with reference to FIGS. 6 and 7 (refer to FIG. 1 as appropriate).
FIG. 6 is a flowchart showing the operation of the trajectory image synthesis apparatus 1 according to the present invention. FIG. 7 is a flowchart showing the flow of processing of the video object extraction operation according to the present invention.

  First, the trajectory image synthesizing device 1 inputs camera images (at least two consecutive video frames) obtained by photographing the subject with a camera and camera parameters by the subject movement start detection means 2 (step S1), and based on the camera parameters. Then, it is determined whether or not the subject has started moving (step S2). If the subject has not started moving (No in step S2), the process returns to step S1, and the camera image and camera parameters of the next image frame are input.

  Further, when the trajectory image synthesizing apparatus 1 detects the start of movement of the subject by the subject movement start detection means 2 (Yes in step S2), it is input to the video storage means 11 and the camera parameter storage means 12 in step S1. The camera image and camera parameters of the image frame are stored (step S3). Here, the trajectory image composition device 1 stores the frame number or time of the video frame at which the subject starts moving in the camera parameter storage unit 12.

  Next, the trajectory image synthesizing device 1 determines whether or not the subject has approached the sky by the subject ascent determination means 3 (step S4). If not yet reached (No in step S4), the camera of the next video frame is determined. The video and camera parameters are input by the subject movement start detection means 2 (step S5), and the process returns to step S3 to the video storage means 11 and the camera parameter storage means 12 and the camera video of the video frame input in step S5. Each camera parameter is stored (step S3). Further, when the subject rise determination unit 3 determines that the subject has approached the sky (Yes in step S4), the trajectory image composition device 1 uses the video object extraction operation described later from the video frame input in step S5. A video object is extracted to calculate a three-dimensional position coordinate (step S6). Here, the trajectory image composition device 1 stores the frame number or time of the video frame at which the subject has approached the sky in the camera parameter storage unit 12.

  In the video object extraction operation, as shown in FIG. 7, the trajectory image synthesis device 1 uses the search position update setting unit 4 to calculate the two-dimensional predicted position calculated in step S8 for the initial condition or the previous video frame. Based on the landing determination result in step S9, a search range of the video object is set (step S61), and the threshold for the luminance of the video object extracted by the threshold determination means 5 is higher than the background image, or The lower one is determined and the foreground area is extracted (step S62).

  Then, the trajectory image composition device 1 uses the object area selection unit 6 to calculate the prediction curve calculated in step S9 in FIG. 6 in the previous video frame and the image of the video object stored in the subject image feature storage unit 13. The area of the video object is selected using the characteristics (step S63), and the three-dimensional position coordinates of the video object are calculated by the object three-dimensional position calculation means 10 (step S64).

  Next, the trajectory image synthesizing apparatus 1 determines whether or not the extraction of all video objects up to the video frame up to the still point is terminated by the search position update setting unit 4 or the three-dimensional predicted position calculation unit 71 Whether or not the extraction of the video object is to be ended is determined based on whether or not the velocity of the ball included in the state quantity X calculated by the means 9 is 0 or less than a threshold value. Alternatively, it is manually determined whether or not the video object has been extracted (step S7).

  The trajectory image composition device 1 indicates the position of the video object in the ball area selected in step S63 in FIG. 7 by the two-dimensional predicted position calculation means 9 if it has not been completed (No in step S7). Based on the coordinates and area, the two-dimensional predicted position of the video object is calculated (step S8). Then, the trajectory image synthesis apparatus 1 predicts the three-dimensional predicted position of the subject in the real space based on the coordinates and area indicating the position of the video object by the prediction curve calculation means 7, calculates the prediction curve, and determines the landing determination means. The landing is determined by 8 (step S9). Subsequently, the trajectory image synthesizing apparatus 1 inputs the camera video and camera parameters of the next video frame by the subject movement start detection unit 2 and stores them in the video storage unit 11 and the camera parameter storage unit 12 (step S10). ), Returning to step S6, the video object extraction operation is executed for the video frame input in step S10.

  If the extraction of the video object after the video object has reached the sky is finished (Yes in step S7), the trajectory image synthesis device 1 uses the search position update setting unit 4 to perform the video storage unit 11 and the camera parameter storage unit. 12 and the video frame stored in step S3 and the camera parameters (including the frame number or time of the video frame where the subject has approached the sky) are read out (step S11). Here, the search position update setting unit 4 determines the video signal of the video frame constituting the video from the frame number or time of the video frame determined by the subject rise determination unit 3 that the subject (video object) has approached the sky. Read the time series in the reverse order in the past time direction. And the locus | trajectory image synthesis apparatus 1 performs video object extraction operation | movement (step S12; step S121 of FIG. 7-step S124).

  Then, the trajectory image synthesis apparatus 1 determines whether or not the extraction of the video object has been completed for all the video frames stored in the video storage unit 11 by the search position update setting unit 4 (step S13), and ends. If not (No in step S13), the two-dimensional prediction position calculation means 9 performs two-dimensional prediction of the video object based on the coordinates and area indicating the position of the video object in the ball area selected in step S123 of FIG. The position is calculated (step S14). Then, the trajectory image composition device 1 predicts the three-dimensional predicted position of the subject in the real space based on the information on the position of the video object by the prediction curve calculation means 7 and calculates the prediction curve (step S15).

Next, the trajectory image synthesizing apparatus 1 returns to step S11, and the search position update setting unit 4 similarly converts the camera image and the camera parameter of the video frame from the video storage unit 11 and the camera parameter storage unit 12. The video frame at time t-1 is read in the reverse order of the time series (step S11), and the video object extraction operation is executed.

  If the extraction of the video object is completed (Yes in step S13), the trajectory image composition device 1 can extract the video object in the video frame when the subject (ball) starts moving (at the time of the shot). It will be. As a result, the trajectory image composition device 1 performs all of the video from the extraction of the video object at the start of the movement of the subject (at the time of the shot) to the rise of the subject to the sky and the landing on the ground and the stationary point. Image coordinates indicating the movement trajectory of the object can be extracted.

  Subsequently, the trajectory image composition device 1 inputs the object three-dimensional position information calculated in step S64 and step S124 of FIG. 7 from the object three-dimensional position storage means 14 by the trajectory image generation means 15, and stores the camera parameters. The camera parameters stored in step S3 and step S10 are input from the means 12, and a trajectory image of the video object is generated (step S16). Furthermore, the trajectory image composition device 1 inputs the trajectory image generated in step S16 by the trajectory image composition means 16, and inputs the video frames stored in step S3 and step S10 from the video storage means 11 and superimposes them. As a result, a composite image including the trajectory image of the ball is acquired (step S17), and the operation ends.

[Configuration of Trajectory Image Synthesizer (Second Embodiment)]
With reference to FIG. 2, a configuration of a video object trajectory image synthesis device according to the second exemplary embodiment of the present invention will be described. FIG. 2 is a block diagram showing an overall configuration of a video object trajectory image synthesis apparatus according to the second embodiment of the present invention.
As shown in FIG. 2, the trajectory image composition device 1B includes a search position update setting unit 4B, a threshold value determining unit 5, an object region selecting unit 6, a prediction curve calculating unit 7, a landing determining unit 8, a two-dimensional display. Predicted position calculation means 9, object three-dimensional position calculation means 10, video storage means 11B, camera parameter storage means 12B, subject image feature storage means 13, object three-dimensional position storage means 14, and trajectory image generation means 15 and trajectory image synthesis means 16. Since the configuration other than the search position update setting unit 4B, the video storage unit 11B, and the camera parameter storage unit 12B is the same as that of the trajectory image synthesis apparatus 1 described with reference to FIG. The description is omitted.

  The video object trajectory image synthesizing device 1B includes a video signal (camera video) obtained by photographing a subject that is launched into the sky and moving with a camera, and camera parameters including information on the posture and angle of view of the camera when the subject is photographed. The video object in the video frame constituting the video is extracted, the three-dimensional position indicating the position of the subject in the real space is obtained, and the trajectory image of the video object in the video frame is generated. This is to synthesize camera video.

  The search position update setting unit 4B reads the video signal in the video frame from the video storage unit 11B in the reverse order of the time series, reads the camera parameters corresponding to the video signal from the camera parameter storage unit 12B, and Object) search range. The search position update setting unit 4B functions as the search position update setting unit 4 (FIG. 1) only by reading the input video frames and camera parameters from the video storage unit 11B and the camera parameter storage unit 12B. Are the same. The video signal read here, the camera parameter, and the search range are output to the threshold value determination means 5.

  The video storage unit 11B stores video signals of video frames taken by the camera in time series, and is configured by a general storage unit such as a hard disk. The video signal of the video frame stored here is read out in the reverse direction by the search position update setting unit 4B and the trajectory image synthesis unit 16 with respect to time series.

  The camera parameter storage unit 12B stores camera parameters related to the posture state of the video frame photographed by the camera in time series, and is configured by a general storage unit such as a hard disk. The camera parameters stored here are read out in the reverse direction by the search position update setting unit 4B and the trajectory image generation unit 15 with respect to time series.

  As described above, the trajectory image composition device 1B goes back in time in the reverse order to the past direction, reads the video signal of the video frame in the video signal, and the corresponding camera parameter, and performs the same process as in the first embodiment. By extracting the video object, it is possible to generate a trajectory curve indicating the movement trajectory of the subject that is reached at the time of the tee shot when the subject starts moving first, and to combine it with the camera video.

  As described above, the trajectory image synthesis device 1 and the trajectory image synthesis device 1B of the video object according to the present invention can automatically extract the hit point P1 at the time of a tee shot, which has been difficult until now, In the automatic setting of the brightness threshold of the area and the calculation of the trajectory curve (see FIG. 4), the accuracy of the ball search method and the ground trajectory extraction after landing (see FIG. 5B) are not limited to the automatic extraction of the landing point P2. ), The ball trajectory curve from the ground to the sky, the landing on the ground, and the stationary point P3 is continued, following the ball tee shot (hit point P1). The operation was fully automated so that it could be extracted with high accuracy. As a result, compared to the conventional example (see FIG. 5A), it is possible not only to shorten the time for extracting the trace display image, but also to avoid erroneous extraction, improve the quality of the trace image, and improve the service. .

  Furthermore, the trajectory image synthesis device 1 is not limited to synthesizing the trajectory image of the ball in golf, but when the background brightness changes abruptly between the infield and the outfield as in baseball and softball (the infield is brown) Even if the outfield is green) and the ball flies linearly without flying up (liner, etc.), a luminance histogram is generated for each video frame, and the ball is moved according to the frequency of the luminance histogram. By changing the luminance threshold value to be extracted, the balls can be extracted continuously, and the trajectory image of the balls can be drawn and synthesized.

1 is a block diagram illustrating an overall configuration of a video object trajectory image synthesis device according to a first embodiment of the present invention. FIG. It is a block diagram which shows the structure of the locus | trajectory image synthesis apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of a threshold value determination means, a prediction curve calculation means, and a landing determination means. It is a display example which shows the trajectory curve of the video object which concerns on embodiment of this invention. (A) It is a display example which shows the locus | trajectory of the video object by a conventional system. (B) It is a display example which shows the locus | trajectory of the video object which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process of the locus | trajectory image synthesis apparatus which concerns on the 1st and 2nd embodiment of this invention. It is a flowchart which shows the flow of a process of the video object extraction operation | movement which concerns on the 1st and 2nd embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1B locus | trajectory image synthesis apparatus 2 Subject movement start detection means 3 Subject rise determination means 4, 4B Search position update setting means 5 Threshold determination means 6 Object region selection means 7 Prediction curve calculation means 8 Landing determination means 9 Two-dimensional prediction position calculation Means 10 Object three-dimensional position calculation means 11, 11B Video storage means 12, 12B Camera parameter storage means 13 Subject image feature storage means 14 Object three-dimensional position storage means 15 Trajectory image generation means 16 Trajectory image synthesis means 51 Luminance histogram generation means 52 Luminance threshold value setting means 53 Foreground / background separation means 71 Three-dimensional predicted position calculation means 72 Screen coordinate conversion means 73 Trajectory curve interpolation means 81 Subject height determination means 82 Landing time determination means 83 Coordinate conversion means

Claims (5)

The movement trajectory of the video object corresponding to the subject is calculated from the video obtained by photographing the moving subject by the camera and the camera parameters including information on the posture and angle of view of the camera when the subject is photographed. A trajectory image synthesis device for a video object that synthesizes a trajectory image representing a moving trajectory of the video object and a video shot by the camera,
Search position update setting means for setting a search range of the video object in a video frame constituting the video;
Generating a luminance histogram which is a luminance histogram of the video frame, and based on the luminance histogram, a threshold value for the luminance for extracting the video object from the search range set by the search position update setting unit; A threshold value determining means for determining whether to extract a region having a luminance higher than a threshold value or a region having a lower luminance, and extracting an image of the determined region as a foreground image including the video object;
Object region selection means for selecting the video object from the foreground image extracted by the threshold value determination means using image characteristics of the video object;
A two-dimensional prediction position for calculating a two-dimensional prediction position indicating the position of the video object in the video frame next to the video frame from the position in the video frame of the video object selected by the object region selection means A calculation means;
Based on the position in the video frame of the video object selected by the object area selection means and the camera parameter, a three-dimensional position indicating the position in the real space of the subject at the time when the video frame was shot is calculated. An object three-dimensional position calculating means,
Object 3D position storage means for storing the 3D position obtained by the object 3D position calculation means;
A trajectory image generating means for generating the trajectory image based on the three-dimensional position stored in the object three-dimensional position storage means;
A trajectory image synthesis means for synthesizing and outputting the trajectory image generated by the trajectory image generation means and the video frame;
The trajectory image of a video object, wherein the search position update setting means sets the search range of the next video frame based on the two-dimensional predicted position calculated by the two-dimensional predicted position calculating means. Synthesizer. Corresponding to the video object in the plurality of video frames input subsequent to the video frame from the position in the video frame of the video object selected by the object area selection means and the camera parameter. 3D predicted position calculating means for sequentially predicting a 3D predicted position indicating the position of the subject in real space;
Screen coordinate conversion means for projecting and transforming the three-dimensional predicted position obtained by the three-dimensional predicted position calculating means to calculate the position of the video object corresponding to the subject in the next video frame;
Trajectory curve interpolation means for interpolating a sequence of points indicating the position in the video frame obtained by the screen coordinate conversion means using a curve,
The object area selecting means is based on the distance from the curve generated by the trajectory curve interpolation means to each area that is a candidate for the video object included in the foreground image of the next video frame. The video object trajectory image composition apparatus according to claim 1, wherein a video object is selected. Subject height determining means for determining whether or not the height of the subject in real space is lower than a preset height based on the three-dimensional predicted position obtained by the three-dimensional predicted position calculating means; With
The search position update setting means includes the search that is wider than the search range of the video frame before the video frame in the video frame next to the video frame at the time when it is determined that the subject height determination means falls below The trajectory image composition device for a video object according to claim 2, wherein a range is set. Video storage means for storing the video;
Camera parameter storage means for storing the camera parameters,
The search position update setting means inputs the time-series video frames constituting the video stored in the video storage means in reverse order, sets the search range in the video frames, and sets the object 3 4. The three-dimensional position is calculated by inputting the time-series camera parameters stored in the camera parameter storage unit in reverse order, and calculating the three-dimensional position. The trajectory image synthesis apparatus for video objects described in 1. A three-dimensional position indicating the position of the subject in real space is calculated from an image obtained by photographing the moving subject with a camera and camera parameters including information on the posture and angle of view of the camera when the subject is photographed. The trajectory image that is stored in an external storage device and generates a trajectory image representing the movement trajectory of the video object corresponding to the subject based on the three-dimensional position, and the video imaged by the camera. To synthesize a computer,
Search position update setting means for setting a search range of the video object in a video frame constituting the video;
Generating a luminance histogram which is a luminance histogram of the video frame, and based on the luminance histogram, a threshold value for the luminance for extracting the video object from the search range set by the search position update setting unit; A threshold value determining means for determining whether to extract a region having higher brightness than a threshold value or to extract a lower region, and extracting an image of the determined region as a foreground image including the video object;
Object region selection means for selecting a video object from the foreground image extracted by the threshold value determination means using image characteristics of the video object;
Two-dimensional predicted position calculating means for calculating a two-dimensional predicted position indicating the position of the video object in the next video frame from the position in the video frame of the video object selected by the object region selecting means;
Based on the position in the video frame of the video object selected by the object area selection means and the camera parameter, a three-dimensional position indicating the position in the real space of the subject at the time when the video frame was shot is calculated. An object three-dimensional position calculation means for storing in the storage device,
A trajectory image generating means for generating the trajectory image based on the three-dimensional position stored in the storage device;
Function as trajectory image synthesis means for synthesizing and outputting the trajectory image generated by the trajectory image generation means and the video frame;
The trajectory image composition program for a video object, wherein the search position update setting means sets the search range of the video frame based on the two-dimensional predicted position calculated by the two-dimensional predicted position calculation means.

Images