JP2007226761A - Locus image composition device for image object, locus image display device for image object, and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a composite image, in which a movement locus of an image object is superimposed on a camera image, by using one camera. <P>SOLUTION: This locus image composition device is provided with an object extraction means 4 acquiring an image coordinate of the image object corresponding to an object from an image frame photographing the moving object, an object position calculation means 5 converting the image coordinate into a sight line vector based on a camera parameter in photographing the image frame, an object position storage means 6 storing a group of sight line vectors acquired from a series of image frames photographing the movement locus, a locus image generation means 7 generating a locus image in the image coordinate corresponding to the camera parameter during photographing based on the group of sight line vectors, and a locus image composition means 10 combining the locus image and the camera image together. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a trajectory image composition device for a video object that shoots a moving subject with a camera to create a trajectory image and combines the trajectory image with the video captured by the camera, and a trajectory image display of the video object that displays the composite image. The present invention relates to a device and a program thereof.

  In a sports program or the like, when a camera captures a subject at a telephoto position, it may be difficult to convey the overall movement of the subject to the viewer. For example, in the first shot of a short hole in a golf broadcast program, the video of a camera installed near the green is generally broadcast, but the golf ball is small and can hardly be confirmed in a wide-angle video, so shooting while chasing the ball with telephoto Is done. Although this telephoto image shows the partial movement of the ball, it is difficult to grasp the overall ball trajectory such as what course the ball has flew to and the height of the ball.

Therefore, in recent sports broadcast programs, various virtual images are produced based on camera posture information such as camera pan, tilt and zoom. For example, in golf relay, the distance between the ball and the cup is displayed on the green, or a distance line is drawn by virtual CG (computer graphics) at the point of the first hit, making it difficult to convey only with camera images. The device is designed to make the feeling easier to understand.
As a device for displaying the trajectory of a ball, Non-Patent Document 1 uses two cameras to follow a hit ball, and based on the inner angle of a triangle having the hit ball and two cameras as vertices and the distance between the two cameras. Based on the principle of triangulation, the three-dimensional position of the hit ball is calculated at an appropriate time interval. And the apparatus which displays the locus | trajectory of the ball | bowl in flight in real time, and displays a flight distance and the remaining distance to a cup is disclosed.
Patent Document 1 discloses a method for calculating the three-dimensional position coordinates of a flying ball based on the triangulation principle based on the positions of two cameras, the optical axis direction, and the angle of view.
Further, in Patent Document 2, using radio wave radars installed at a plurality of locations in a golf course, the radio wave is applied to the golf ball, the flight speed and distance of the ball are obtained from the Doppler frequency and phase difference of the reflected wave, and the triangle A method for calculating the three-dimensional position coordinates of a ball by the principle of surveying is disclosed.

Japanese Patent Laying-Open No. 2003-42716 (paragraphs 0032 to 0034, FIG. 3) JP-A-8-266701 (paragraphs 0039 to 0040, FIG. 3) Yamauchi, Kato, Noguchi, Abe, Minami, “3D position visualization of flying object trajectory display system“ Shot View ”, Information Processing Society of Japan, Vol.1995, No.063, pp31-36

  However, when used in a relay program, it is desirable not to exceed the range of existing relay facilities. That is, it must be operated without adding a new sensor camera or photographer. In general, golf relay is operated with one virtual camera per hole. In the ball position measurement method according to Non-Patent Document 1 and Patent Document 1, since shooting of a ball by two cameras is necessary, two cameras are required and two people are required to operate each camera. A photographer is required. Further, in the ball position measurement method disclosed in Patent Document 2, it is necessary to install special devices called radio wave radar devices at a plurality of locations on a golf course.

  Therefore, in the present invention, in view of such a problem, an image in which a moving trajectory of a small video object that can be operated by one camera and it is difficult to grasp the overall movement is superimposed on the camera video is synthesized. An object of the present invention is to provide a trajectory image synthesis device for a video object, a trajectory image display device for a video object that displays the synthesized image, and a program thereof.

  The present invention has been made to achieve the above object, and the video object trajectory image composition device according to claim 1, when a moving subject is photographed by a camera and a video signal and the subject is photographed. A trajectory of a video object corresponding to the subject is measured from camera parameters including information on the posture and angle of view of the camera, a trajectory image representing the trajectory of the measured video object, and a video shot by the camera Is an image object trajectory image synthesizing apparatus comprising an object extracting means, an object position calculating means, an object position storing means, a trajectory image creating means, and a trajectory image synthesizing means.

According to such a configuration, the video object trajectory image composition device includes a video signal when a moving subject is photographed and a camera posture when the subject is photographed, that is, a pan angle / tilt angle indicating a photographing direction of the camera, And camera parameters including information such as the zoom amount relating to the angle of view. For example, when shooting a moving trajectory of a small subject such as a golf ball, the cameraman sets the angle of view (zoom amount on the telephoto side) of the subject so that the position of the video object corresponding to the subject can be accurately measured. Then follow the shooting.
Then, the trajectory image composition device extracts a video object from each captured video frame using an image processing method in order to measure the movement trajectory of the video object corresponding to the subject such as a golf ball by the object extraction unit, The two-dimensional coordinates (image coordinates) of the video object in the video frame are acquired.

  Then, the trajectory image synthesizing device determines the position of the camera based on the camera parameters when the image frame obtained by acquiring the image coordinates of the image object obtained by the object extraction means is captured by the object position calculating means. A two-dimensional line-of-sight vector as an origin, that is, a position coordinate indicating the viewing direction of the video object viewed from the camera position is converted. In this conversion, first, based on the angle of view information, a displacement from the center of the video frame that is the shooting direction (optical axis direction) of the camera is calculated as image coordinates. Then, by adding the position coordinates (line-of-sight vector) represented by camera posture information (pan angle / tilt angle) indicating the shooting direction of the camera and the displacement, the accurate position coordinates (line-of-sight vector) of the video object are added. Can be obtained.

The line-of-sight vector of the video object acquired in this way is stored in the object position storage means. By sequentially performing such line-of-sight vector acquisition processing on a series of video frames obtained by shooting a moving subject, information on a series of trajectory points of the video object is obtained as a line-of-sight vector group in the object position storage means. be able to.
Then, the trajectory image composition device reads the line-of-sight vector group stored in the object position storage means by the trajectory image creation means, and uses the CG (computer graphics) method to generate a video to be synthesized with the trajectory image. A trajectory image at the image coordinates corresponding to the captured camera parameter is created. Thereafter, a trajectory image created by CG is superimposed on the camera video to be synthesized by the trajectory image synthesizing means to generate a synthesized image and output it as a video.

  The video object trajectory image composition apparatus according to claim 2 is the video object trajectory image composition apparatus according to claim 1, wherein the trajectory image creation means includes a trajectory curve generation means, an image coordinate conversion means, And a locus image drawing means.

According to this configuration, the trajectory image synthesizing apparatus for the video object reads the line-of-sight vector group stored in the object position storage unit and generates the trajectory curve by the trajectory curve generation unit of the trajectory image generation unit.
Then, the trajectory image synthesizing apparatus performs a Bezier curve, a spline curve, 2 on the line-of-sight vector group of the video object acquired as a discrete trajectory point for moving at high speed by the trajectory curve generating means of the trajectory image creating means. A curve such as a secondary curve is applied as a trajectory curve. By applying these curves, the locus points obtained by the measurement are interpolated to calculate a line-of-sight vector group of the locus constituting points constituting the locus curve.

Then, the trajectory image synthesis device corresponds to the camera parameters (pan angle / tilt angle / view angle) when the video to be superimposed with the trajectory image is captured by the image coordinate conversion unit of the trajectory image generation unit. Is converted into an image coordinate group.
Furthermore, the trajectory image composition device uses the trajectory image creation means of the trajectory image creation means to set conditions such as the color, line type, thickness, and animation display of the trajectory image based on the data of the image coordinate group constituting the trajectory point. In consideration, a trajectory image to be superimposed is created.

  According to a third aspect of the present invention, there is provided the video object trajectory image synthesizing apparatus according to the first aspect, wherein the trajectory image creating means includes a trajectory curve generating means and a trajectory composing point storage means. And image coordinate conversion means and trajectory image drawing means.

  According to this configuration, the trajectory image synthesizing apparatus for the video object temporarily stores the line-of-sight vector group of the trajectory constituent points generated by the trajectory curve generating means in the trajectory constituent point storage means. The trajectory image composition device reads the eye vector group from the trajectory composing point storage means and uses it when converting the image coordinate group corresponding to the camera parameters of the video on which the trajectory image is superimposed by the image coordinate conversion means. When the camera parameter of the video on which the trajectory image is to be superimposed is changed, the image coordinate conversion means reads the line-of-sight vector group from the trajectory constituent point storage means again, and converts the line-of-sight vector group of the trajectory constituent point into the image coordinate group. The locus image drawing means draws a locus image drawing.

  The video object trajectory image composition apparatus according to claim 4 is the video object trajectory image composition apparatus according to any one of claims 1 to 3, further comprising an object designation unit. did.

  According to such a configuration, the trajectory image composition device for a video object designates a desired point as a trajectory point in an arbitrary video frame by the object designation unit, and acquires image coordinates in the video frame. The acquired image coordinates of the designated point are converted into a line-of-sight vector by the object position calculation means based on the camera parameters when the video frame used for the designation is photographed. The line-of-sight vector at the designated point is additionally stored in the line-of-sight vector group acquired by the object extraction unit and stored in the object position storage unit.

  The video object trajectory image composition apparatus according to claim 5 is the video object trajectory image composition apparatus according to any one of claims 1 to 4, wherein the object position calculation means is a two-dimensional object. An image coordinate represented by orthogonal coordinates is converted into a line-of-sight vector represented by two-dimensional polar coordinates.

  According to such a configuration, the trajectory image composition device for a video object captures a video frame obtained by acquiring the image coordinates represented by the two-dimensional orthogonal coordinates obtained by the object extracting means by the object position calculating means. Based on the angle-of-view information included in the camera parameters at that time, it is converted into two-dimensional polar coordinates in the video frame. The pan angle / tilt angle indicating the optical axis direction (shooting direction) of the camera corresponds to the horizontal coordinate (horizontal angle) / vertical coordinate (vertical angle) of the two-dimensional polar coordinate in the direction of the center of the image frame to be shot. Thus, the line-of-sight vector of the video object can be calculated by adding the pan angle and tilt angle to the horizontal and vertical coordinates of the two-dimensional polar coordinates in the video frame of the video object, respectively.

  The video object trajectory image composition device according to claim 6 is the video object trajectory image composition device according to any one of claims 1 to 5, further comprising a position predicting means, drop inspection, and the like. An output means and a coordinate conversion means are provided.

According to such a configuration, the video object trajectory image synthesis apparatus sequentially predicts the three-dimensional position of the subject based on the image coordinates and the size of the video object for each video frame by the position predicting unit. For this prediction, for example, an extended Kalman filter can be used. Then, the trajectory image synthesizing apparatus drops the subject onto the ground based on the three-dimensional position predicted by the position predicting unit by the drop point detecting unit and the three-dimensional spatial coordinates determined in advance based on the camera position. Detect points. That is, the fall point detection means determines that the subject has fallen to the ground when it has reached the coordinate plane of the ground determined in advance in three-dimensional space coordinates.
The trajectory image synthesizer converts the drop point detected by the drop point detection means into two-dimensional image coordinates in the video frame based on the camera parameters when the video frame is captured by the coordinate conversion means. To do. Thereby, the trajectory image composition device can detect the drop point of the subject and synthesize it as a trajectory image.

  The video object trajectory image composition device according to claim 7 is the video object trajectory image composition device according to claim 6, further comprising subject movement start detection means, wherein the position prediction means detects subject movement start detection. When the start of the movement of the subject is detected by the means, the prediction of the position of the subject is started.

According to this configuration, the trajectory image composition device for the video object uses the subject movement start detection unit to move the subject based on the difference between the change amount of the camera parameter of the camera capturing the subject and the predetermined threshold value. Detect the start of That is, when the photographer takes a picture of the subject, the direction of the camera changes greatly when the subject starts to move. Therefore, it is determined that the subject has started to move when the amount of change becomes larger than a predetermined threshold. .
The trajectory image synthesizing apparatus predicts the three-dimensional position of the subject after the video frame in which the subject is determined to have started to move by the position predicting unit.

  According to an eighth aspect of the present invention, there is provided a trajectory image display device for a video object, wherein the trajectory image composition device according to any one of the first to seventh aspects and a composite image synthesized by the trajectory image composition device. And image display means for displaying video.

  According to this configuration, it is possible to create a trajectory image of a video object and display a video in which the trajectory image of the video object is superimposed on a video shot by the same camera with only one camera operation.

  The video object trajectory image display program according to claim 9 is a camera parameter including a video signal when a moving subject is photographed by a camera, and information on an attitude and angle of view of the camera when the subject is photographed. In order to measure the trajectory of the video object corresponding to the subject and to display the synthesized trajectory image representing the trajectory of the measured video object and the video shot by the camera, the object is extracted. It is made to function as a means, an object position calculation means, a locus image creation means, and a locus image synthesis means.

  According to such a configuration, the trajectory image display program of the video object receives the video signal when the moving subject is photographed and the camera parameters including the camera posture information and the angle-of-view information at the time of photographing. The means extracts a video object corresponding to the subject using an image processing method for each video frame, and acquires image coordinates in the video frame of the extracted video object. The acquired image coordinates are converted into a line-of-sight vector based on the camera parameters by the object position calculation means. The line-of-sight vector obtained by this conversion is stored in the object storage means.

  Then, the trajectory image display program obtains the line-of-sight vector of the video object from the series of video frames in which the moving subject is photographed by the object extracting unit, the object position calculating unit, and the object position storing unit. A series of line-of-sight vectors is obtained. Then, the trajectory image creating means creates a trajectory image at the image coordinates of the video frame corresponding to the camera parameter when the camera video to be superimposed is captured based on the line-of-sight vector group, and the trajectory image synthesizing means. Thus, the created trajectory image is output as a composite image superimposed on the camera image.

According to the first aspect of the present invention, the video object trajectory image synthesizing apparatus measures the trajectory of the video object corresponding to the subject from the video of the subject shot by one camera, and the camera shot from the same location. Since a trajectory image corresponding to the camera parameter is created and combined with the video, a composite image video in which the trajectory image is superimposed on the camera image can be obtained by using only one camera.
In addition, if the video that superimposes the trajectory image is the camera video that is currently being shot, a trajectory image that matches the camera video that is being shot can be created and displayed in a superimposed manner. This provides an excellent effect that the trajectory image can be displayed without forcing the movement of the viewpoint.

  According to the second aspect of the present invention, since a trajectory curve is generated based on the measured trajectory point, even if there is a video frame in which the trajectory point measurement has failed, a smooth trajectory curve is obtained by interpolating between the video frames. be able to.

  According to the third aspect of the present invention, since the line-of-sight vector group of the trajectory constituent points generated by the trajectory curve constituent means is stored in the trajectory constituent point storage means, the camera parameters of the video on which the trajectory image is to be superimposed are changed. In such a case, the line-of-sight vector of the trajectory composing point can be read again from the trajectory composing point storage means and used, and the amount of calculation necessary for creating the trajectory image can be reduced. A trajectory image can be quickly created.

  According to the fourth aspect of the present invention, since the trajectory point is added by the object specifying means, the object extraction means, for example, because the background is complicated such as a hitting point / falling point / resting point of a golf ball. Thus, even for trajectory points for which stable automatic extraction is difficult, a moving trajectory can be added after shooting, and a trajectory image without interruption can be created.

According to the fifth aspect of the present invention, the object position calculation means converts the image coordinates represented by the two-dimensional orthogonal coordinates into the line-of-sight vector represented by the two-dimensional polar coordinates, and therefore converts the image coordinates from the orthogonal coordinates to the polar coordinates. By simply adding the pan angle and tilt angle, the line-of-sight vector of the video object can be calculated.
In addition, since the line-of-sight vector is expressed in polar coordinates, the position coordinates can be expressed even when the shooting direction of the camera is directed to any direction in the sky, and a trajectory image can be generated even when the subject moves over a wide range. Can do.

  According to the invention described in claim 6, it is possible to detect the fall of the subject on the ground by predicting the position of the subject in the three-dimensional space. For this reason, this detection time can be used as the timing for generating the trajectory image, and the time required for generating the trajectory image can be shortened as compared with the case where the start of the generation of the trajectory image is manually instructed.

  According to the seventh aspect of the present invention, it is possible to start predicting the position of the subject based on the movement of the camera. For this reason, it is possible to detect the start of movement of the subject more reliably than in the case where the timing for starting the prediction is set manually.

  According to the invention described in claim 8 or claim 9, the trajectory is measured from the video of the video object shot by one camera, and the camera video shot from the same place is determined according to the camera parameter. Since a trajectory image is created and displayed in a superimposed manner, a video in which a trajectory image is superimposed on a camera video can be displayed by using only one camera.

The best mode for carrying out the invention will be described below with reference to the drawings. Here, a case where a moving subject is a golf ball and a trajectory image of a golf ball (video object) photographed by a camera is combined with a video and displayed is described as an example.
[First Embodiment]
(Principle of the present invention)
First, the principle of the present invention will be described.
When displaying the trajectory of a video object, the three-dimensional position coordinates of the moving subject (golf ball) are generally required as in the conventional technique described above. The coordinate system may be rectangular or polar. However, when considering the polar coordinates with the camera as the origin, the pan angle and tilt angle of the camera are the same as the horizontal and vertical angles of the optical axis of the camera. Convenient because it corresponds. However, it is assumed that the camera is in an ideal state where the intersection of the pan axis and tilt axis of the camera coincides with the optical principal point of the lens. If the depth information to the remaining golf ball is known, it becomes a three-dimensional polar coordinate, but the distance to the golf ball cannot be calculated only by information from one camera.

  The two-dimensional polar coordinates (line-of-sight vector) of horizontal and vertical angles are lines on the three-dimensional coordinates. However, it is displayed as a point on the camera image obtained by measuring the line-of-sight vector. Therefore, if the camera that measures the position of the subject is the same as the camera that shoots the video that superimposes the trajectory (the camera is installed at the same location), the trajectory is displayed even if the depth information to the subject cannot be obtained. Is possible.

(Configuration of trajectory image display device (including trajectory image composition device))
Next, the configuration of a video object trajectory image display device (including a trajectory image synthesis device) according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing the overall configuration of a video object trajectory image display apparatus including a video object trajectory image composition apparatus according to the first embodiment of the present invention. Here, after describing the camera 3 used in the trajectory image display device 2, the configuration of the trajectory image display device 2 will be described.

(camera)
The camera 3 includes a camera body 3a having a camera lens 3b having a zoom function, and a camera platform 3c that supports the camera body 3a so as to be independently rotatable in the horizontal and vertical directions. The camera 3 continuously outputs a pan angle / tilt angle, which is posture information related to the shooting direction of the camera, and a zoom amount, which is information related to the angle of view of the camera, together with the captured video signal.

Here, the camera 3 detects the horizontal rotation angle and the vertical rotation angle of the camera platform supporting the camera body 3a by a rotary encoder (not shown) provided in the rotation unit of each camera platform 3c, Output as pan and tilt angles. Instead of this rotary encoder, a gyro sensor is mounted on the camera body 3a to detect angular velocities around the horizontal and vertical axes of the camera operation, calculate the rotation angle by integrating the detected angular velocities, and The angle / tilt angle may be obtained.
The camera 3 outputs a zoom amount in accordance with the setting of the focal length of the camera lens 3b. The camera 3 includes a camera lens having a zoom function, and can output posture information indicating a shooting direction of the camera 3 and information corresponding to an angle of view value such as a zoom amount in addition to a video signal. Any form can be used.

(Configuration of locus image display device)
Next, the configuration of the trajectory image display device will be described. The trajectory image display device 2 includes a video signal input from the camera 3, camera posture information (pan angle / tilt angle) when the subject is photographed, and information (zoom amount) regarding the camera angle of view. Based on the parameters, the trajectory of the subject (here, a golf ball) taken by the camera 3 is synthesized and displayed on the image. Here, the trajectory image display device 2 includes an object extraction unit 4, an object position calculation unit 5, an object position storage unit 6, a trajectory image creation unit 7, an object designation unit 8, and an angle of view information storage unit 9. , A trajectory image synthesis means 10 and an image display means 11 are provided.

  The object extraction means 4 extracts a video object corresponding to the subject from the video frame using an image processing method in the video signal obtained by photographing the subject with the camera 3. Note that the coordinates (image coordinates) of the video object extracted by the object extraction unit 4 are output to the object position calculation unit 5 together with the camera parameters corresponding to the video frame from which the image coordinates are acquired. However, the object extracting unit 4 corresponds to the zoom amount by referring to the view angle information indicating the relationship between the zoom amount and the view angle value stored in advance in the view angle information storage unit 9 for the zoom amount in the camera parameters. The angle of view of the video frame to be converted is output to the object position calculation means 5.

  Here, the configuration of the object extracting means 4 will be described in detail with reference to FIG. FIG. 2 is a block diagram showing the configuration of the object extraction means. Here, the object extraction unit 4 includes a chroma key processing unit 41, a filter processing unit 42, and a centroid calculation unit 43.

  The chroma key processing means 41 extracts video object candidates from the video frame by chroma key processing. For example, when the video object is a flying golf ball, the chroma key processing unit 41 extracts pixels constituting the video object using the difference between the color of the golf ball and the background sky color. In this example, video object candidates are extracted by chroma key processing, but other extraction methods such as frame difference extraction may be used.

  The filter processing means 42 extracts a predetermined video object from the video object candidates extracted by the chroma key processing means 41. For example, the filter processing unit 42 uses a shape characteristic of a video object (in the case of a golf ball, it is a sphere and thus has a circular shape on a planar image). The video object is extracted by performing shape recognition processing. Note that the filter processing means 42 may extract a video object by color, brightness, etc. in addition to the shape.

  The center-of-gravity calculating unit 43 calculates the center of gravity of the image of the video object extracted by the filter processing unit 42 and detects the position (image coordinate) of the video object in the video frame.

  Here, with reference to FIG. 3 and FIG. 4 (refer to FIG. 2 as appropriate), a method for extracting a video object from a video frame and detecting a position will be described. FIG. 3 is a diagram for explaining a method of extracting a video object and detecting a position from a video frame. FIG. 4 is a diagram showing a state where a golf ball is extracted from a video frame.

  FIG. 3A is a view showing a state of a video frame in which a golf ball in flight is photographed against the sky. The object extraction unit 4 uses the fact that the background is a blue sky as a first stage, and the chroma key processing unit 41 extracts video object candidates. As a second step, the filter processing means 42 performs filter processing for extracting the color, brightness, and shape of the golf ball on the extracted video object candidate, and selects a desired video object from video object candidates such as trees having different characteristics. A golf ball is extracted. An extracted image of the video object OBJ extracted from the video frame in this way is shown in FIG.

  FIG. 3C shows the relationship between the video object OBJ and the coordinate axes of the video frame. Here, the image coordinate system is orthogonal coordinates in which the center of the video frame is the origin and the x-axis and y-axis are set in the horizontal and vertical directions, respectively. Since the extracted video object has an area rather than a point, the center of gravity of the extracted image is specified by the center of gravity calculation means 43 to determine the center position of the video object, and the two-dimensional image coordinates in the video frame are detected. To do. Then, as shown in FIG. 3D, the image coordinates (200, 300) of the video object OBJ in this image coordinate system can be acquired.

When the object extracting means 4 extracts a golf ball as a video object from a golf tee shot video, as shown in FIG. 4, a series of video frames F _A , F _{B1 to} F _B17 , F _C , F _D The golf balls are extracted sequentially. Note that point A is a hit point, point B (B1 to B17) is a point during flight, point C is a drop point, and point D is a stationary point.
Returning to FIG. 1, the description of the configuration of the trajectory image display device will be continued.

The object position calculation means 5 is a golf ball (subject) based on image coordinates input from the object extraction means 4 or an object designation means 8 to be described later and a pan angle / tilt angle and an angle of view as camera posture information. A two-dimensional line-of-sight vector (positional coordinates) with the origin at the camera position representing the detection direction is calculated. The line-of-sight vector (position coordinates) calculated by the object position calculation means 5 is stored in the object position storage means 6.
Here, the object position calculation means 5 adds a new line-of-sight vector to the line-of-sight vector group stored in the object position storage means 6 in accordance with the insertion destination designated by the object designation means 8 described later.

Note that the coordinate system of the line-of-sight vector may be orthogonal coordinates, polar coordinates, or another coordinate system, but when considering the polar coordinates with the camera position as the origin, the camera pan angle and tilt angle are the same as the optical axis of the camera. This is convenient because it corresponds to the direction, that is, the horizontal and vertical angles which are the coordinate values of the center point of the video frame. Therefore, in the present embodiment, two-dimensional polar coordinates with the camera position as the origin are used as the coordinate system of the line-of-sight vector.
Here, the object position calculation means 5 converts the image coordinates represented by two-dimensional orthogonal coordinates based on the angle-of-view value into an error angle represented by two-dimensional polar coordinates, that is, the horizontal / distance from the image center of the video frame to the position of the video object. The angle of view is converted into a vertical angle, and the error angle (horizontal / vertical angle) is added to the pan angle and tilt angle, respectively, to calculate a line-of-sight vector represented by two-dimensional polar coordinates.

Here, the configuration of the object position calculation means 5 will be described in detail with reference to FIG. FIG. 6 is a block diagram showing the configuration of the object position calculation means. Here, the object position calculation unit 5 includes an error angle calculation unit 51 and an addition unit 52.
The error angle calculation means 51 converts image coordinates into error angles based on the field angle value. Specifically, the error angle calculation means 51 uses the horizontal pixel number of the video frame as WID and the horizontal coordinate of the video object (two-dimensional orthogonal coordinates (image coordinates) with the image center of the video frame as the origin). x, when the field angle in the horizontal direction is FOV, calculated by the following equation horizontal error angle ρ _x (1).

The adding means 52 calculates the line-of-sight vector by adding the error angle converted by the error angle calculating means 51 and the pan angle / tilt angle.
Specifically, as shown in the following equation (2), the adding unit 52 adds the error angle ρ _x converted by the error angle calculating unit 51 and the pan angle ρ _cam of the camera, thereby adding the camera. The horizontal angle ρ _obj of the video object on the polar coordinates with the position of as the origin is calculated.

When it is assumed that the video object is always at the center of the image, x is always “0” in the above example. Therefore, since ρ _{x is} also “0”, the horizontal angle of the video object is equal to the pan angle of the camera. That is, if the video object is always at the center of the image, the pan angle can be used as it is. The vertical angle of the video object can also be obtained by the same calculation by using the tilt angle.
As described above, the line-of-sight vector displayed on the two-dimensional polar coordinates of the video object (horizontal and vertical angles with the camera position as the origin) can be obtained.
Returning to FIG. 1, the description of the configuration of the trajectory image display device will be continued.

The object position storage means 6 stores the line-of-sight vector (position coordinates) calculated by the object position calculation means 5, and is a general storage device such as a hard disk.
The object position storage means 6 stores the coordinate data of each line-of-sight vector and the number in the trajectory point sequence in association with each other in order to hold time-series information. When a line-of-sight vector is added by the object designating unit 8 to be described later, the number of the additional point is set as the insertion destination number, and the numbers for the subsequent line-of-sight vectors are sequentially lowered and stored in the object position storage unit 6. .
However, in the case of a still image in which only the measured trajectory points are plotted as the trajectory image to be superimposed on the camera video, the time series information between the line-of-sight vectors is unnecessary, so it is not necessary to assign a number or the like to store it. .

The object position storage unit 6 stores a line-of-sight vector group corresponding to a plurality of trajectories for each trajectory. The data of these line-of-sight vectors is read out by the trajectory image creation means 7.
In this way, by storing the line-of-sight vector group in the object position storage unit 6, the line-of-sight vector group can be corrected using a correction unit such as the object designating unit 8 described later. Further, even if the scene (camera parameter) of the video on which the trajectory image is to be superimposed is changed, the trajectory image can be recreated as appropriate according to the camera parameter. Furthermore, it is possible to read a line-of-sight vector group corresponding to a plurality of loci and draw a plurality of loci simultaneously.

  The trajectory image creating means 7 creates a trajectory image corresponding to the camera parameter when the video to be superimposed is taken based on the line-of-sight vector group of the video object read from the object position storage means 6. is there. Here, the trajectory image creation means 7 includes a trajectory curve generation means 71, a trajectory composing point storage means 72, an image coordinate conversion means 73, and a trajectory image drawing means 74.

The trajectory curve generation means 71 reads a series of line-of-sight vector groups corresponding to one trajectory from the object position storage means 6, and applies a curve function such as a spline curve, a B-spline curve, a Bezier curve, or a quadratic curve to the line-of-sight vector group. By applying this, a trajectory curve is defined, and a line-of-sight vector group of a point sequence constituting the trajectory curve is calculated.
Here, a case where a trajectory curve is generated using a cubic Bezier curve as a curve function will be described. The cubic Bezier curve is a curve expressed by an equation using two end points and two control points using one variable t, and the end points are P ₀ and P ₃ , and the control points are P ₁ and P. When ₂ is expressed by the following equation (3).

  Here, by applying the line-of-sight vector of the object to equation (3) four points at a time, arbitrary points between the respective end points can be calculated, and a smooth trajectory can be drawn. Accordingly, the trajectory curve generation means 71 samples (calculates) several tens of line-of-sight vector groups from the defined curve function, and stores them in the trajectory constituent point storage means 72 as a line-of-sight vector group of trajectory constituent points. The line-of-sight vector group of the constituent points of the trajectory curve calculated by the trajectory curve generating means 71 is temporarily stored in the trajectory constituent point storage means 72 in a state managed for each trajectory curve.

  The trajectory constituent point storage means 72 stores a line-of-sight vector group of constituent points of the trajectory curve calculated by the trajectory curve generating means 71, and is a general storage device such as a hard disk. The line-of-sight vector group stored in the locus composing point storage means 72 is read by the locus image creation means 7.

  The image coordinate conversion means 73 converts the coordinate values of the line-of-sight vector group read from the trajectory constituent point storage means 72 from polar coordinates to the image coordinate group corresponding to the camera parameter currently being photographed. . It should be noted that the zoom amount in the camera parameters is used by referring to the view angle information stored in advance in the view angle information storage means 9 to be described later, and converting it to the view angle value of the video frame corresponding to the zoom amount. To do.

Here, the configuration of the image coordinate conversion means 73 will be described in detail with reference to FIG. FIG. 8 is a block diagram showing the configuration of the image coordinate conversion means. Here, the image coordinate conversion means 73 includes a subtraction means 73a and an image coordinate calculation means 73b.
The subtracting means 73a calculates an error angle by subtracting the pan angle / tilt angle from the line-of-sight vector. Here, the subtracting unit 73a subtracts the pan angle / tilt angle of the camera currently being photographed from the line-of-sight vector for each point for all points of the trajectory composing points, and the video frame in which the trajectory composing points should superimpose the trajectory image. An error angle indicating how many times the image center is shifted in the horizontal and vertical directions is calculated.
Specifically, the subtracting unit 73a uses the following formula (4) when the horizontal angle of one point of the locus constituting point group of interest is ρ _obj , the pan angle is ρ _cam , and the horizontal error angle is ρ _sub. An error angle ρ _sub in the horizontal direction is calculated.

The image coordinate calculation means 73b converts the error angle calculated by the subtraction means 73a into image coordinates based on the angle of view value.
Specifically, the image coordinate calculation means 73b is a video frame on which the trajectory image of the trajectory composing point is to be superimposed by the following equation (5), where the angle of view value is FOV and the number of pixels in the horizontal direction of the image is WID. The x coordinate of the position to be drawn inside is calculated.

In this way, the x coordinate of the position on the video frame on which the trajectory image of the trajectory composing point of interest is to be superimposed can be obtained. Note that the y-coordinate can be obtained by calculating the vertical direction in the same manner. By calculating this for all trajectory composing points, it is possible to obtain an image coordinate group on the video frame currently being photographed.
Returning to FIG. 1, the description of the configuration of the trajectory image display device will be continued.

The trajectory image drawing means 74 draws a trajectory image (CG) using the trajectory composing point group (image coordinates) calculated by the image coordinate conversion means 73. The trajectory image drawing means 74 draws a continuous trajectory curve by connecting the points of the trajectory constituent point group to each other.
The configuration of the trajectory image creation means 7 has been described above. When the trajectory image to be superimposed on the camera video is a still image in which only the measured trajectory points are plotted, the trajectory curve generation means 71 and the trajectory constituent point storage means 72 are used. Is unnecessary. In addition, in an apparatus that creates only a still image of a trajectory point measured as a trajectory image, when there is a change in the camera parameter of the camera video on which the trajectory image is superimposed, the line-of-sight vector group is read from the object position storage means 6; The read line-of-sight vector group is converted into an image coordinate group corresponding to the camera parameter by the image coordinate converting unit 73 of the trajectory image creating unit 7, and each point of the image coordinate group is appropriately selected by the trajectory image drawing unit 74 such as a circle or a triangle. What is necessary is just to comprise so that it may draw (plot) as a simple figure.

The object designating unit 8 additionally designates a trajectory point of a video object that could not be automatically detected by the object extracting unit 4. That is, the object designating means 8 manually designates and adds the position of the video object in a scene that is difficult to extract, and the number of designated points or the extracted point sequence successfully extracted by the object extracting means 4 is added. On the other hand, the insertion destination of the designated point is arbitrarily designated.
Note that the object designation unit 8 includes, as information on the additionally designated locus point, an insertion destination in the line-of-sight vector group stored in the object position storage unit 6, image coordinates, and a video referred to when the locus point is designated. The camera parameter corresponding to the frame is output to the object position calculation means 5. However, the object designating unit 8 refers to the angle-of-view information stored in the angle-of-view information storage unit 9 in the same way as the object extracting unit 4 for the zoom amount in the camera parameters, and the video corresponding to the zoom amount. The image is converted into a frame angle of view value and then output to the object position calculation means 5.

Here, with reference to FIG. 7 (refer to FIG. 1 as appropriate), the configuration of the object specifying means 8 will be described in detail. FIG. 7 is a block diagram showing the configuration of the object specifying means. The object specifying means 8 can be realized using a PC (personal computer).
Here, the object designation unit 8 includes an image storage unit 81, a camera parameter storage unit 82, an insertion destination designation unit 83, an object position designation unit 84, and a camera parameter acquisition unit 85.

The image storage unit 81 records (stores) a video of a moving subject and is a general storage device such as a hard disk.
By storing the video here, even if it is difficult to recognize the video object from the video frame by tracing the video frame in the forward direction in time, The position of the ball that is the object can be recognized.

  The camera parameter storage unit 82 stores camera parameters corresponding to an image obtained by photographing a moving subject, and is a general storage device such as a hard disk.

  The insertion destination designating unit 83 designates the insertion destination of the locus point to be added to the locus point group (line-of-sight vector group) of the video object extracted by the object extracting unit 4. The insertion destination designation means 83 receives a designation of the insertion destination by an operator (cameraman) via an input means (not shown) such as a PC keyboard, and adds a trajectory to be added to the already obtained line-of-sight vector group. The point insertion destination is output to the object position calculation means 5.

  The object position designation means 84 designates the image coordinates of the locus point to be added with reference to the video frame. The object position specifying means 84 displays a desired video frame on the display device of the PC, causes the mouse to function as a pointing device, and superimposes a cursor linked to the operation of the mouse on the screen on which the video frame is displayed. When a button provided on the mouse is pressed, the coordinates on the video frame at the point where the cursor is displayed are output to the object position calculation means 5 as image coordinates. In addition to the mouse, the pointing device may be a touch pad, a trackball, or the like. Alternatively, a touch panel may be provided on the display screen, and a point on the screen may be directly touched to indicate.

  The camera parameter acquisition unit 85 acquires the camera parameter corresponding to the video referred to when the image coordinates are designated from the camera parameter storage unit 82 or the camera 3. That is, the camera parameter acquisition unit 85 acquires the camera parameter from the camera parameter storage unit 82 when using the video frame stored in the image storage unit 81, and uses the camera frame when using the video frame of the currently captured camera video. Directly obtain the camera parameters output from.

Here, with reference to FIG. 4 and FIG. 5 (refer to FIG. 7 as appropriate), a method for designating an insertion point in the object designation means 8 will be described.
When the operator selects “spot”, the object designating unit 8 inserts the point designated by the insertion destination designating unit 83 as the first point of the line-of-sight vector group constituting the trajectory. When the operator selects “still point”, the insertion destination designating unit 83 adds the point designated as the last point of the line-of-sight vector group.

  For example, in FIG. 4, in the state where B1 to B17 are obtained as the trajectory points in flight, when adding “batting points”, the operator designates the insertion destination by instructing “before B1”. In addition, when adding “falling point”, the operator designates the insertion destination by instructing “after B17”. Then, when adding the “still point”, the operator can designate the insertion destination as the last point by instructing “after C”, after adding the “falling point”. It is also possible to specify an insertion destination in a trajectory point during flight. As described above, the number of points to be specified and the insertion destination can be arbitrarily set.

FIG. 5 shows an example in which the trajectory points B1 to B17 in flight successfully extracted by the object extracting means 4 (FIG. 1) and the trajectory image of the hit point A already added by the object specifying means 8 are displayed together with their numbers. Is shown. In this case, the operator designates “falling point” as the locus point to be added next, and designates the position in the video displayed on the screen by an input means such as a mouse. For example, the object designation means 8 displays a cross-shaped cursor on the screen by the object position designation means 84 and moves the cursor to a desired position by operating the mouse. Thereafter, the image coordinates of the “falling point” are determined by pressing a button provided on the mouse.
The video frame displayed in FIG. 5 is a wide-angle video that can display the entire movement trajectory. However, after specifying the insertion destination while referring to the wide-angle video, the desired telephoto image is displayed and the position of the trajectory point is indicated. It may be specified.

Although the function for adding a locus point in the object designating unit 8 has been described here, a function for deleting an unnecessary locus point may be added. When deleting an unnecessary trajectory point, the trajectory point sequence can be displayed on the screen of the PC together with its number, and the number of the trajectory point to be deleted can be designated from the keyboard.
Returning to FIG. 1, the description of the configuration of the trajectory image display device will be continued.

  The angle-of-view information storage means 9 stores in advance the relationship between the zoom amount and the angle-of-view value in the camera parameters, and is a general storage device such as a hard disk. For example, if the zoom end is “0.0” at the wide end and “1.0” at the tele end, the zoom amount between “0.0” and “1.0” The angle value is stored. For example, when the zoom amount is “0.1”, the angle of view value is “50.0”.

  The trajectory image synthesizing unit 10 draws a trajectory image on a trajectory image frame memory (not shown) based on the trajectory image data input from the trajectory image creation unit 7. The trajectory image drawn in the trajectory image frame memory is combined with a video signal obtained by photographing the trajectory image by the camera 3 and output as a video signal of the composite image.

The image display means 11 displays the video signal in which the trajectory image is synthesized by the trajectory image synthesis means 10. Specifically, it is a display device such as a CRT (cathode ray tube) monitor or a liquid crystal display monitor.
Although the configuration of the trajectory image display device 2 has been described above, the trajectory image synthesis device 1 that generates a trajectory image can be configured by omitting the image display unit 11 from the trajectory image display device 2.

  The video object trajectory image synthesis device 1 and the video object trajectory image display device 2 can be implemented by creating a dedicated hardware part or all of them, but executing a general computer program, It can also be realized by operating an arithmetic device, a storage device, an input device, an image display device and the like in the computer. This program (video object trajectory image display program) can be distributed via a communication line, or can be written on a recording medium such as a CD-ROM for distribution.

(Operation of the trajectory image display device (trajectory image composition device))
Next, referring to FIG. 11 (see FIG. 1 as appropriate), the operation of the video object trajectory image display device 2 including the video object trajectory image synthesis device 1 will be described. FIG. 11 is a flowchart showing a processing flow of the video object trajectory image display apparatus including the video object trajectory image synthesis apparatus.
The trajectory image display device 2 inputs the video signal (video frame) and camera parameters input from the camera 3 to the object extraction unit 4 for each video frame (step S10). Then, the object extraction unit 4 extracts a predetermined video object from the input video frame, and acquires the two-dimensional coordinates (image coordinates) of the video object in the video frame (step S11). Then, the object extraction unit 4 outputs the acquired image coordinates to the object position calculation unit 5 together with camera parameters corresponding to the video frame from which the image coordinates are acquired. However, the zoom amount in the camera parameters is input to the object position calculation means 5 after being converted into the corresponding field angle value.

  Further, the trajectory image display device 2 uses the line-of-sight vector based on the image coordinates input from the object extraction unit 4 by the object position calculation unit 5 and the pan angle / tilt angle and the angle of view as the posture information of the camera 3. Is calculated (step S12), and the calculated line-of-sight vector (object position) is stored in the object position storage means 6 (step S13). Here, when the calculation of the line-of-sight vector of the video object from one video frame is completed, the object extraction unit 4 checks whether there is a next video frame (step S14). If there is a next video frame (Yes in step S14), the process returns to step S10, the video signal of the next video frame and the corresponding camera parameter are input, and the processing from video object extraction to line-of-sight vector calculation / storage is performed. (Step S10 to Step S13) are repeated.

  On the other hand, when the acquisition of the line-of-sight vector of the video object corresponding to the subject is completed from a series of video frames obtained by photographing the moving subject (No in step S14), the object specifying means 8 adds a trajectory point of the video object. Whether or not (step S15). If there is an additional trajectory point of the video object (Yes in step S15), the object specifying unit 8 acquires information on the trajectory point that is specified and added by the operator with reference to an arbitrary video frame (step S16). .

  Then, the trajectory image display device 2 uses the object designation means 8 as the information about the additionally designated trajectory point, the insertion destination in the line-of-sight vector group stored in the object position storage means 6, the image coordinates, and the trajectory point. The camera parameter corresponding to the video frame referred to when it is designated is output to the object position calculation means 5. However, the zoom amount in the camera parameters is converted to the angle value of the video frame corresponding to the zoom amount by referring to the angle information stored in advance in the angle angle information storage unit 9 and then the object position calculating unit 5. Is input.

  Then, the trajectory image display device 2 calculates the line-of-sight vector by the object position calculation means 5 based on the image coordinates input from the object designation means 8 and the pan angle / tilt angle and the angle of view, which are camera posture information. Then, the calculated line-of-sight vector (object position) is additionally stored in the line-of-sight vector group of the object position storage means 6 in accordance with the insertion destination in the viewpoint vector group designated by the object designation means 8 (step S18). Then, when the trajectory image display device 2 finishes storing the object position of the designated trajectory point, the trajectory image display device 2 further checks whether or not a trajectory point of the video object has been added (step S15). (Yes in step S15), the processing from step S16 to step S18 is repeated. If no locus point is added (No in step S15), locus image creation processing is performed by the locus image creation means 7 (step S19).

  When the designated point is no longer added (No in step S15), the trajectory image display device 2 uses the trajectory image creation unit 7 based on the line-of-sight vector group of the video object read from the object position storage unit 6. A trajectory image corresponding to the camera parameter when the video to be superimposed is photographed is created (step S20). The detailed operation of the trajectory image creation means 7 will be described later.

  Subsequently, the trajectory image display device 2 synthesizes the trajectory image created by the trajectory image creation means 7 by the trajectory image synthesis means 10 and the video signal currently being photographed by the camera 3 to obtain a video signal of the composite image. Output (step S20).

  Then, the trajectory image display device 2 converts the video signal obtained by synthesizing the trajectory image output by the trajectory image synthesizing unit 10 into an image display unit such as a CRT (cathode ray tube) monitor or a liquid crystal display monitor as necessary. 11 (step S21). The video signal combined with the trajectory image is transmitted via a television broadcast wave, an Internet line, or the like, and is displayed on a display device on the receiver side such as a television receiver or a PC (personal computer).

Next, the subroutine of the flowchart shown in FIG. 11 will be described in detail.
<Object extraction step>
Details of the object extraction step S11 will be described with reference to FIG. Here, FIG. 12 is a flowchart showing the processing flow of the object extracting means.
First, the object extraction unit 4 extracts video object candidates from the input video frame by the chroma key processing unit 41 as a first step (step S30). Then, as a second stage, the filter processing means 42 performs filter processing on the extracted video object candidates, and extracts video objects from the video object candidates extracted by the chroma key processing means 41 (step S31). Then, the center of gravity of the video object extracted by the filter processing means 42 is calculated by the center of gravity calculating means 43 to identify the center position of the video object, and the two-dimensional image coordinates of the video object in the video frame are determined. Obtain (step S32).

<Object position calculation step>
Next, details of the object position calculating step (S12 and S17) will be described with reference to FIG. Here, FIG. 13 is a flowchart showing the processing flow of the object position calculation means.
First, when the image coordinates of the video object and the camera parameters are input by the object position calculation unit 5 (step S40), the trajectory image display device 2 converts the zoom amount of the camera parameters into the zoom amount and the field angle value. The angle-of-view value is obtained by referring to the angle-of-view information storage means 9 that stores the relationship in advance (step S41). Then, the trajectory image display device 2 converts the image coordinates into an error angle based on the field angle value by the error angle calculation unit 51 of the object position calculation unit 5 (step S42). Then, the addition means 52 of the object position calculation means 5 adds the horizontal and vertical error angles calculated by the error angle calculation means 51 to the pan angle and tilt angle values, respectively, thereby obtaining the line-of-sight vector of the video object. Calculate (step S43).

<Object specification step>
Next, details of the object specifying step S16 will be described with reference to FIG. Here, FIG. 14 is a flowchart showing the processing flow of the object designating means.
First, the trajectory image display device 2 is added to the line-of-sight vector group already obtained by the designation by the operator through the input means such as the PC keyboard by the insertion destination designation means 83 in the object designation means 8. The locus point insertion destination is acquired (step S50).

  When the insertion destination is designated, the object designating unit 8 instructs the object position designating unit 84 to refer to a video frame according to the operator's selection in order to identify the position of the locus point (step S51). . As the video frame to be referred to, a desired video frame is read out from a series of video frames in which the target video object is captured, which is recorded in the image storage unit 81 with reference to the currently captured video frame. Can be selected.

In addition, when referring to the video recorded in the image storage unit 81, it is preferable to refer to the video frame in which the video object to be added is captured. For example, if you want to add a drop point, the object position specifying means 84, acquires the video frame F _C obtained by photographing the falling point from the image storage unit 81 in FIG. 4, is displayed on the display device of the PC (step S52) . At this time, when frame advance is selected by the operator, the object position specifying unit 84 reads and displays the video frames recorded in the image storage unit 81 one by one, so that the operator specifies the desired video frame accurately. (Repeat step S51 to step S53).
When the trajectory image display device 2 acquires a desired video frame by the object position specifying unit 84 and displays it on the PC screen (Yes in step S53), the trajectory image display device 2 moves the cursor in conjunction with the mouse operation by the operator. Let Then, the coordinates on the video frame of the cursor display point when the mouse button is pressed by the operator are acquired as the image coordinates of the locus point to be added (step S54).

When the image coordinates of the locus point to be added are designated, the camera parameter acquisition unit 85 acquires camera parameters corresponding to the referenced video frame (step S55).
When the referenced video frame is a video recorded in the image storage unit 81, the camera parameter acquisition unit 85 acquires the camera parameter corresponding to the referenced video frame from the camera parameter storage unit 82.
In addition, when using the camera video currently being shot as the video frame to be referenced, the camera parameter acquisition unit 85 directly acquires the camera parameter output from the camera 3.

  Further, the image coordinates of the point to be added as the trajectory point acquired by the object designating unit 8, the camera parameters, and the information on the insertion destination for the line-of-sight vector group are output to the object position calculating unit 5 and the line-of-sight vector is calculated ( Step S17 in FIG. 11 is additionally stored in the line-of-sight vector group stored in the object position storage means 6 (step S18 in FIG. 11).

<Track image creation step>
Next, details of the locus image creation step S19 will be described with reference to FIG. Here, FIG. 15 is a flowchart showing the flow of operation of the trajectory image creation means.
First, the trajectory image display device 2 reads a series of line-of-sight vector groups corresponding to the trajectory from the object position storage means 6 by the trajectory curve generation means 71 of the trajectory image creation means 7 and uses the Bezier curves and spline curves as the line-of-sight vector groups. A trajectory curve is defined by applying a curve function such as a quadratic curve, and a line-of-sight vector group of the point sequence constituting the trajectory curve is calculated (step S60). The calculated line-of-sight vector group is temporarily stored in the trajectory composing point storage means 72 (step S61).

Then, the trajectory image display device 2 reads the data of the line-of-sight vector group stored in the trajectory composing point storage means 72 by the image coordinate conversion means 73 of the trajectory image creation means 7 and obtains the line-of-sight vector group represented by polar coordinates. Then, it is converted into an image coordinate group corresponding to the camera parameter currently being photographed (step S62).
Further, the trajectory image display device 2 draws a trajectory image based on the trajectory constituent point group (image coordinates) calculated by the image coordinate conversion means 73 by the trajectory image creation means 74 of the trajectory image creation means 7 (step S63). ).

  Then, the trajectory image display device 2 confirms whether or not another trajectory is to be drawn by the trajectory image creating means 7 (step S64). If there is a trajectory to be further drawn (Yes in step S64), the trajectory image display device 2 draws the image. A trajectory constituent point group of the power trajectory is specified (step S66), and a conversion process (step S62) and a trajectory image drawing process (step S63) from the line-of-sight vector group read from the trajectory constituent point storage means 72 to the image coordinate group are performed. Do it sequentially. On the other hand, when there is no other trajectory to be drawn (No in step S64), the trajectory image creating means 7 confirms whether there is a change in the camera parameter during shooting of the video on which the trajectory image is to be superimposed (step S65). ). If the camera parameter has been changed (Yes in step S65), the trajectory constituent point is read from the trajectory constituent point storage means 72 for the trajectory to be drawn, and a trajectory image corresponding to the changed camera parameter is drawn again ( Step S62 to Step S64, Step S66). On the other hand, if the camera parameter has not been changed (No in step S65), the process is terminated, and the video image captured by the camera 3 and the trajectory image are combined (step S20 in the flowchart shown in FIG. 11).

Through the above operation, the video object trajectory image synthesis device 1 outputs a video signal of a composite image obtained by synthesizing the video object trajectory image and the camera video, and the video object trajectory image display device 2 A composite image can be displayed.
As a result, the trajectory image display device 2 (trajectory image synthesis device 1) generates a trajectory image (CG) and combines it with the video, so that the trajectory of the subject as shown in FIG. It can be displayed (synthesized).
Further, since the locus image is CG, the color, shape, size, line type, etc. of the locus can be arbitrarily designated. For example, as shown in FIG. 10, if the color is changed for each player in a golf tee shot (the line type is changed in the figure), the difference in the trajectory of the player can be clearly shown. The trajectory image display device 2 (trajectory image composition device 1) can also display a moving image by gradually drawing from the hit point to the stationary point.

In the present embodiment, the video on which the trajectory image is superimposed has been described by taking the video currently being shot by the camera as an example. However, the video is not limited to the live video being shot by the camera, but a hard disk, an optical disc, The present invention can also be applied to a video recorded on a storage means such as a video tape as a configuration in which camera parameters when the video is shot can be stored and read.
In the present embodiment, a trajectory curve is created by applying a curve function such as a Bezier curve based on the line-of-sight vector group of trajectory measurement points acquired by the object extracting means 4 and the object specifying means 8. Although the constituent points are recalculated and drawn, only the trajectory measurement points may be drawn (plotted) without applying the curve function.
Also, it is not limited to images taken using commercial television cameras, but can be applied to images taken using webcams installed indoors or in urban areas, as long as it is an imaging means with pan, tilt and zoom functions. can do.
In this embodiment, the ball trajectory at the golf relay has been described as an example, but the present invention can be applied to other competitions such as baseball and soccer. Further, the present invention can be applied not only to a competition but also to a moving object such as a person or a car for displaying a trajectory of the moving object that can be extracted from a video frame.

[Second Embodiment]
(Principle of the present invention)
In the first embodiment described above, after the acquisition of the line-of-sight vector is completed by the object designating unit 8 of FIG. 1, information on the locus point is added. This additional work can be started quickly if the timing when the acquisition of the line-of-sight vector is completed can be detected in real time.
Also, when adding a ball drop point, the work time can be shortened by estimating the drop time and drop position.
Therefore, here, it is assumed that the timing when the subject falls on the ground and its point are detected.

(Configuration of locus image display device)
First, the configuration of a video object trajectory image display device (including a trajectory image synthesis device) according to the second embodiment of the present invention will be described with reference to FIG. FIG. 16 is a block diagram showing the overall configuration of a video object trajectory image display device including a video object trajectory image synthesis device according to the second embodiment of the present invention.
As shown in FIG. 16, the trajectory image display device 2B includes an object extraction unit 4B, an object position calculation unit 5, an object position storage unit 6, a trajectory image creation unit 7, an object designation unit 8, and field angle information. A storage unit 9, a trajectory image synthesis unit 10, an image display unit 11, and an object drop position prediction unit 12 are provided. Since the configuration other than the object extraction unit 4B and the object drop position prediction unit 12 is the same as that of the trajectory image display device 2 described with reference to FIG. 1, the same reference numerals are given and description thereof is omitted.

The object extraction unit 4B extracts the two-dimensional coordinates (image coordinates) and the size (area) of the video object for each video frame. Here, the configuration of the object extraction unit 4B will be described with reference to FIG. 17 (refer to FIG. 16 as appropriate). FIG. 17 is a block diagram showing the configuration of the object extraction means.
As shown in FIG. 17, the object extracting unit 4B includes a chroma key processing unit 41, a filter processing unit 42, and a centroid calculating unit 43B. Since the configuration other than the center-of-gravity calculating unit 43B is the same as that of the object extracting unit 4 described with reference to FIG. 2, the same reference numerals are given and description thereof is omitted.

The center-of-gravity calculating unit 43B calculates the coordinates (image coordinates) of the center of gravity of the video object extracted by the filter processing unit 42, and outputs it together with the area of the video object. That is, the center of gravity calculating means 43 (FIG. 2) outputs only the image coordinates of the center of gravity, and the center of gravity calculating means 43B outputs the area of the video object together. The image coordinates and area of the center of gravity are output to the object drop position prediction means 12 (FIG. 16). Note that the other output destinations of the image coordinates of the center of gravity are the same as the output destination described in the center of gravity calculation means 43 (FIG. 2).
Returning to FIG. 16, the description of the configuration of the trajectory image composition device 1B will be continued.

The object fall position prediction means 12 is based on the image coordinates and area of the center of gravity of the video object extracted by the object extraction means 4B, and the fall position (predicted fall position) of the ball that is the subject extracted as the video object on the ground. Is to predict. The fall position predicted by the object fall position prediction means is output to the object designation means 8 as drop point designation information.
Here, the configuration of the object drop position prediction unit 12 will be described with reference to FIG. 18 (refer to FIG. 16 as appropriate). FIG. 18 is a block diagram showing the configuration of the object drop position detection means.
As shown in FIG. 18, the object drop position prediction unit 12 includes a shot detection unit 121, a position prediction unit 122, a drop point detection unit 123, and a coordinate conversion unit 124.

  The shot detection means (subject movement start detection means) 121 detects the start of movement of the subject based on the difference between the camera parameter change amount of the camera 3 that is photographing the subject (golf ball) and a predetermined threshold value. To do. Here, since the subject is a golf ball, the shot detection means 121 detects the timing at which the golf ball was hit. When the ball is hit, the cameraman suddenly raises the camera 3 in order to catch the ball in the video. At this time, the tilt angle of the camera parameter changes abruptly.

Therefore, here, the shot detection means 121, for each unit time (video frame), the tilt angle theta _t of the video frame at a certain time t, immediately before (t-1) of the video frame of the tilt angle theta _t-1 As shown in the following formula (6), when a difference from the above becomes larger than a predetermined threshold T, it is detected that the golf ball has been hit. The shot predicting means 121 notifies the position predicting means 122 that the golf ball has been hit, and the position predicting means 122 starts operating.

The position predicting means 122 predicts the three-dimensional position of the ball as the subject based on the image coordinates and area of the center of gravity of the video object extracted by the object extracting means 4B and the camera parameters at the time of extraction. It is. Here, the position predicting means 122 sequentially estimates the three-dimensional position of the ball using an extended Kalman filter.
Here, as shown in the following equation (7), the state quantity X of the extended Kalman filter is expressed by the position coordinates (x, y, z) of the ball on the three-dimensional orthogonal coordinates, and the velocity (v _x , v _y , v _z), the acceleration _{(a x,} a _y, and _{a z),} the observation amount Y, the image coordinates of the center of gravity of the video objects extracted by the object extraction unit 4B _(X g, and _{Y g)} and area (a) . The superscript T represents the transposition of the matrix.

In addition, here, as shown in the following equation (8), the state transition matrix of one frame unit time and F, were added system noise v _t in parabolic motion under certain gravity g Things were used as exercise models. Note that Q is the covariance matrix of the system noise v _t . Also, I and O are 3 × 3 unit matrix and zero matrix, respectively, and t is the number of frames. E represents an expected value.

Then, the position predicting unit 122 sequentially predicts the position of the ball every unit time according to the state estimation formula shown in the following formula (9). Here, P is an error covariance matrix of state quantities. Incidentally, failed extracted by the object extraction unit 4B (not observed quantities are obtained), the element values of the error covariance matrix P would increase as the value of the covariance matrix Q of system noise v _t.

Here, as shown in the following equation (10), an observation having a covariance matrix R in the perspective projection h (X) with camera parameters (pan angle: φ, tilt angle: θ, focal length: f) What added the noise w _t was used as an observation model. The initial position coordinates (x, y, z) of the ball are determined in advance by calibration before shooting. The initial velocity (v _x , v _y , v _z ) of the ball and the area formula coefficient s are determined in advance from a reference shot trajectory before imaging.

  Here, when the object extraction unit 4B succeeds in the extraction (observation amount is obtained), the position prediction unit 122 performs the state quantity X and the error covariance matrix P according to the update formula shown in the following formula (11). Update elements of. H is an observation sensitivity matrix.

  As described above, the position predicting unit 122 sequentially updates the observation when the observation is obtained based on the state estimation expression shown in the equation (9) and the update equation shown in the equation (11). State estimation is performed to predict the three-dimensional position of the ball.

The drop point detection means 123 detects the drop point of the subject (ball) on the ground based on the three-dimensional position predicted by the position prediction means 122 and the three-dimensional space coordinates determined in advance based on the position of the camera 3. It is to detect.
In other words, the drop point detection means 123 is set as the ground in advance in the three-dimensional space coordinates in which the position of the ball predicted by the position prediction means 122 (the element v _z of the state quantity X) has the position of the camera 3 as the origin. It is determined that the ball has fallen when it falls below the height (z-axis direction). The position of the drop point detected by the drop point detection unit 123 is output to the coordinate conversion unit 124.
The drop point detection means 123 may store the video frame or time at which the drop point is detected in the object position storage means 6. This makes it possible to easily search for an image when the ball falls later.

The coordinate conversion unit 124 converts the drop point (predicted drop position) detected by the drop point detection unit 123 into image coordinates in the video frame based on the camera parameters.
That is, the coordinate conversion means 124 converts the predicted drop position indicated by the three-dimensional orthogonal coordinates into the two-dimensional image coordinates based on the camera parameters (pan angle: φ, tilt angle: θ, focal length: f) of the camera 3. Convert to The image coordinates can be obtained by using a conversion formula similar to the perspective projection h (X) shown in the formula (10). Image coordinates indicating the predicted drop position are output to the object designating unit 8.

As described above, since the trajectory image display device 2B can predict when the ball is dropped and its drop point, the operator (cameraman) can specify the designated work such as insertion of the trajectory point at the stage of detecting the fall. Can start quickly. Further, since the ball can be predicted at the time of dropping and the point of dropping, the image at the time of dropping can be quickly retrieved from the image storage means 81 (see FIG. 7), and the time for specifying the dropping position can be shortened.
Note that the trajectory image display device 2B may superimpose and display the predicted range on the screen as a predicted range corresponding to the element value of the error covariance matrix. Good. For example, as shown in FIG. 19, the predicted fall position V is displayed, and the predicted range W is displayed based on the size of the element value of the error covariance matrix. As a result, when the operator (cameraman) needs to search for the ball within the predicted range W when correcting the drop position, the operator can quickly correct the drop position.
The trajectory image display device 2B can be configured as the trajectory image composition device 1B by omitting the image display means 11.

1 is a block diagram showing an overall configuration of a video object trajectory image display device including a video object trajectory image synthesis device according to a first embodiment of the present invention; FIG. It is a block diagram which shows the structure of an object extraction means. It is a figure for demonstrating a mode that a video object is extracted and a position is detected from a video frame. (A) is an original image of a video frame in which a video object and a background are photographed. (B) is a figure which shows a mode that the video object was extracted from the video frame. (C) is a diagram showing the relationship between the extracted video object and the coordinate axes in the video frame. (D) is a figure which shows the image coordinate of the extracted video object. It is a figure which shows the mode of the to-be-moved subject and the image | video frame which carries out the tracking imaging | photography of the subject. It is a figure which shows a mode that addition of a locus | trajectory point is designated with respect to the locus | trajectory point of the video object measured from the image | video which image | photographed the moving subject. It is a block diagram which shows the structure of an object position calculation means. It is a block diagram which shows the structure of an object designation | designated means. It is a block diagram which shows the structure of an image coordinate transformation means. It is an example of a screen which shows the image | video which superimposed the locus image by the locus | trajectory display apparatus of the video object which concerns on this invention. It is an example of a screen which shows the image | video which superimposed the several locus | trajectory by the locus | trajectory display apparatus of the video object which concerns on this invention. It is a flowchart which shows the flow of a process of the locus | trajectory image display apparatus of the video object containing the locus | trajectory image production apparatus of the video object which concerns on this invention, and the locus | trajectory image synthetic | combination apparatus of a video object. It is a flowchart which shows the flow of a process of an object extraction means. It is a flowchart which shows the flow of a process of an object position calculation means. It is a flowchart which shows the flow of a process of an object designation | designated means. It is a flowchart which shows the flow of a process of a locus | trajectory image creation means. It is a block diagram which shows the whole structure of the locus | trajectory image display apparatus of the video object containing the locus | trajectory image synthesis apparatus of the video object which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the object extraction means which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of an object fall position prediction means. It is an example of a screen which shows the image | video which superimposed the predicted fall position and the prediction range by the locus | trajectory display apparatus of the video object which concerns on this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Trajectory image synthesis apparatus 2 Trajectory image display apparatus 3 Camera 4 Object extraction means 5 Object position calculation means 6 Object position storage means 7 Trajectory image creation means 71 Trajectory curve generation means 72 Trajectory constituent point storage means 73 Image coordinate conversion means 74 Trajectory image Drawing means 8 Object designation means 9 View angle information storage means 10 Trajectory image synthesis means 11 Image display means 12 Object drop position prediction means 121 Shot detection means (subject movement start detection means)
122 Position prediction means 123 Drop point detection means 124 Coordinate conversion means

Claims (9)

The trajectory of the video object corresponding to the subject is measured from the video signal when the moving subject is photographed by the camera and the camera parameters including information on the posture and angle of view of the camera when the subject is photographed. A trajectory image synthesis device for a video object that synthesizes a trajectory image representing a trajectory of a measured video object and a video shot by the camera,
Object extraction means for extracting the video object from the video frame in the video signal and obtaining image coordinates of the video object in the video frame;
Object position calculating means for converting the image coordinates of the video object into a line-of-sight vector having the position of the camera as an origin, based on camera parameters when the video frame from which the video object is extracted is captured;
Object position storage means for storing a line-of-sight vector group composed of a plurality of line-of-sight vectors obtained by the object position calculation means from a series of video frame groups obtained by photographing the moving video object;
Based on the line-of-sight vector group stored in the object position storage means, a trajectory image creation means for creating a trajectory image at image coordinates corresponding to camera parameters when a video to be superimposed is taken.
Trajectory image composition means for outputting a composite image obtained by compositing the trajectory image created by the trajectory image creation means and the video taken by the camera;
A trajectory image synthesis apparatus for video objects, comprising: The trajectory image creating means includes
A trajectory curve generating means for generating a trajectory curve based on a line-of-sight vector group stored in the object position storage means, and calculating a line-of-sight vector group of trajectory composing points constituting the trajectory curve;
Image coordinate conversion means for converting a line-of-sight vector group of trajectory constituent points calculated by the trajectory curve generation means into an image coordinate group corresponding to a camera parameter when a video to be superimposed with a trajectory image is captured;
A trajectory image drawing means for drawing a trajectory image based on the image coordinate group of the trajectory composing points converted by the image coordinate conversion means;
The trajectory image composition device for a video object according to claim 1, comprising: The trajectory image creating means includes
A trajectory curve generating means for generating a trajectory curve based on a line-of-sight vector group stored in the object position storage means, and calculating a line-of-sight vector group of trajectory composing points constituting the trajectory curve;
Trajectory constituent point storage means for storing a line-of-sight vector group of trajectory constituent points generated by the trajectory curve generating means;
Image coordinate conversion means for converting a line-of-sight vector group of trajectory constituent points calculated by the trajectory curve generation means into an image coordinate group corresponding to a camera parameter when a video to be superimposed with a trajectory image is captured;
A trajectory image drawing means for drawing a trajectory image based on the image coordinate group of the trajectory composing points converted by the image coordinate conversion means;
The trajectory image composition device for a video object according to claim 1, comprising: An object designating unit that designates a video object with reference to a video frame photographed by the camera and acquires image coordinates of the designated video object,
Based on the camera parameters when the video frame referred to when the video object is taken, the image coordinates of the video object acquired by the object designating means are determined by the object position calculating means. Convert to the line-of-sight vector as the origin,
Additionally storing in the line-of-sight vector group stored in the object position storage means;
The trajectory image synthesis device for a video object according to any one of claims 1 to 3. The object position calculating means converts image coordinates represented by two-dimensional orthogonal coordinates into a line-of-sight vector represented by two-dimensional polar coordinates;
The trajectory image synthesis apparatus for a video object according to any one of claims 1 to 4, characterized in that: Position prediction means for sequentially predicting the three-dimensional position of the subject based on the image coordinates and size of the video object for each video frame and the camera parameter;
Drop point detection means for detecting a drop point of the subject on the ground based on the three-dimensional position predicted by the position prediction means and three-dimensional space coordinates determined in advance based on the position of the camera;
Coordinate conversion means for converting the drop point detected by the drop point detection means into image coordinates in the video frame, based on camera parameters when the video frame is shot,
The trajectory image synthesis device for a video object according to any one of claims 1 to 5, further comprising: Subject movement start detection means for detecting the start of movement of the subject based on a difference between a change amount of a camera parameter of a camera that is photographing the subject and a predetermined threshold;
The position predicting means starts predicting the position of the subject when the subject movement start detecting means detects the start of movement of the subject;
The trajectory image synthesis apparatus for a video object according to claim 6. A trajectory image synthesis device according to any one of claims 1 to 7,
Image display means for displaying an image of a synthesized image synthesized by the trajectory image synthesizer;
A trajectory image display device for a video object, comprising: The trajectory of the video object corresponding to the subject is measured from the video signal when the moving subject is photographed by the camera and the camera parameters including information on the posture and angle of view of the camera when the subject is photographed. In order to synthesize and display the trajectory image representing the trajectory of the measured video object and the video imaged by the camera,
Object extraction means for extracting the video object from the video frame in the video signal and obtaining image coordinates of the video object in the video frame;
The image coordinates of the video object are converted into a line-of-sight vector with the camera position as the origin based on the camera parameters when the video frame from which the video object was extracted is shot, and stored in the object position storage means Object position calculating means,
Trajectory image creation that creates a trajectory image at image coordinates corresponding to camera parameters when a video to be superimposed with a trajectory image is captured based on a line-of-sight vector group consisting of a plurality of line-of-sight vectors stored in the object position storage means means,
Trajectory image composition means for outputting a composite image obtained by compositing the trajectory image created by the trajectory image creation means and the video photographed by the camera;
A trajectory image display program for video objects, characterized in that

Images