JP4181473B2 - Video object trajectory synthesis apparatus, method and program thereof - Google Patents

Description

  The present invention relates to a video object trajectory synthesizing apparatus, a method thereof, and a program thereof that synthesize a moving trajectory in which a video object included in an input video moves on the video.

Conventionally, there are various video object extraction techniques for extracting and tracking an object (video object) included in a video (for example, Patent Document 1).
Attempts have been made to expand the scope of video production by using the positional information of the extracted video objects to give the video special effects. As an example of giving a special effect to this video, the ball (video object) thrown by the pitcher is tracked in a baseball broadcast or the like, and then the movement trajectory of the ball is displayed on the pitcher's pitch video by CG (Computer Graphics). There is something to synthesize. Thereby, the viewer can more feel the change of the trajectory of the ball, for example, the change ball, from the moment when the pitcher releases the ball until the catcher catches the ball. Conventionally, when drawing this baseball pitching trajectory, the ball is photographed by a plurality of cameras, the trajectory of the ball on three-dimensional coordinates is calculated, and synthesized as a CG video with the pitching video.
JP 2003-44860 (paragraphs 0018 to 0027, FIG. 1)

  The video object extraction technique described in Patent Document 1 described above is to extract and track a video object from a video, and to synthesize a moving trajectory of the video object in real time based on the extracted position information of the video object. Absent. In addition, in order to synthesize the moving trajectory of a video object from the position information of the video object extracted by the video object extraction technology to the original video, a person manually moves to the original video based on the extracted position information. Therefore, there is a problem that it is impossible to synthesize (draw) the movement locus in real time.

  Furthermore, when the trajectory of a ball in a baseball broadcast is synthesized with a pitcher's pitch video, there is a problem that a plurality of cameras are required and the device configuration becomes large. In this case, the ball trajectory is combined with the pitched video as a CG video, and there is a demand to synthesize the moving trajectory using the live-action ball video in order to further enhance the video effect.

  The present invention has been made in view of the above-described problems, and can extract and track a video object from a video, and can synthesize a moving trajectory of the video object on the video in real time and a live action. An object of the present invention is to provide a video object trajectory synthesis apparatus, a method thereof, and a program thereof.

The present invention was created to achieve the above object, and first, the video object trajectory synthesis apparatus according to claim 1 extracts and tracks a video object included in an input video, A video object trajectory synthesizing apparatus that synthesizes a moving trajectory of the video object on the video, the object candidate image generating means, the extraction condition storage means, the object extracting means, the trajectory image generating means, and the search area estimating means And an image composition means.

  According to such a configuration, the video object trajectory synthesizer obtains an object candidate image obtained by extracting video object candidates from frame images in units of still images that are input in time series by the object candidate image generation unit and constitute the video. Generate. For example, by binarizing a frame image according to luminance, or by specifying a region having motion based on a difference between a previously input frame image and a next input frame image, An object candidate image from which a region to be extracted is extracted is generated.

  In the video object trajectory synthesis apparatus, the object extraction unit refers to the extraction condition stored in advance in the extraction condition storage unit, so that the position of the video object to be extracted and the video object are extracted from the object candidate images. Extract feature quantities to be characterized. By storing the extraction conditions of the video object to be extracted in advance in this extraction condition storage means, it becomes possible to specify one video object from a plurality of video object candidates. As a result, the position of the video object to be extracted in the frame image and the unique feature amount of the video object are specified. Here, the position of the video object can use the barycentric coordinates of the video object. In addition, the luminance, color, texture, and the like of the video object can be used as the feature amount of the video object.

Then, the video object trajectory synthesizer generates a trajectory image in which only the moving trajectory of the video object is drawn from the position and the feature amount of the video object extracted by the object extraction means by the trajectory image generation means. The trajectory image can be generated by sequentially overwriting the feature amount (for example, texture) of the video object extracted for each frame image in time series.
The trajectory image generated in this manner is sequentially combined with the frame image by the image combining means, thereby generating a composite video in which the moving trajectory of the video object to be extracted is combined with the video.

Then, the video object trajectory synthesis apparatus is configured to search a search area for searching for a video object in the next input frame image based on the position of the video object extracted by the object extraction means by the search area estimation means. By applying a Kalman filter to the barycentric coordinates of the object, the position of the next video object to be input is predicted and estimated from the prediction.
The search area estimation means updates the Kalman gain, which is the filter sensitivity of the Kalman filter, based on the observation noise covariance that is the covariance of the observation noise and the state covariance that is the covariance of the state quantities in the frame image. and the state covariance is small when successful tracking of video objects, DOO Ru large when the tracking fails.

According to such a configuration, the video object trajectory synthesis device uses the search area estimation unit to determine the search area for the video object in the next input frame image based on the position of the video object extracted by the object extraction unit. It can be set in the vicinity area of the video object. That is, by using the Kalman filter, it is possible to estimate the position of the video object that moves sequentially and limit the search area .

The video object trajectory synthesis apparatus according to claim 2 is the video object trajectory synthesis apparatus according to claim 1, wherein the trajectory image generation means has succeeded in extraction based on the extraction result of the object extraction means. Trajectory interpolation means for interpolating the position of the video object that has failed to be extracted from the position of the video object is provided.

According to such a configuration, the video object trajectory synthesizer maintains the position of the video object that has been successfully extracted by the trajectory image generation means, and if the extraction of the video object has failed, the subsequent extraction succeeded. Thus, based on the position of the video object that has been successfully extracted and the position of the video object that has been previously held, the position that has failed to be extracted is interpolated by interpolation or the like. As a result, a continuous trajectory image of the video object can be generated .

The video object trajectory composition method according to claim 3 , wherein the video object trajectory composition method extracts and tracks a video object included in the input video, and composes a moving trajectory of the video object on the video. The method includes an object candidate image generation step, an object extraction step, a trajectory image generation step, a search area estimation step, and an image composition step.

According to this procedure, in the video object trajectory synthesis method, an object candidate image obtained by extracting video object candidates from frame images in units of still images that are input in time series and constitute a video by the object candidate image generation step. Generate.
Note that the object candidate image is generated from at least one or more images of a difference image obtained by calculating a difference between a luminance image, a contour image, and frame images having different input times in a frame image input in time series.

  For example, if the video object to be extracted has a brightness greater than or equal to a specific brightness (or less), a candidate video object is extracted from a brightness image having only brightness information as a frame image. If the brightness of the video object is significantly different from the brightness of the surroundings, a video object candidate is extracted from the contour image from which only the contour is extracted. Alternatively, if the video object has a large movement, a video object candidate is extracted from a difference image obtained by taking a difference between different frame images. As a result, the amount of calculation can be reduced as compared with the case where the video object is extracted from the color image. Then, after combining the luminance image, the contour image, and the difference image, binarization is performed to generate an object candidate image.

Then, in the object extraction step, the image object trajectory synthesis method extracts the position of the image object to be extracted and the feature quantity characterizing the image object from the object candidate images based on preset extraction conditions.
Here, at least one of the area, luminance, color, aspect ratio, and circularity of the video object to be extracted can be used as the extraction condition. Thereby, even when there are a plurality of video objects in the frame image, the video objects to be extracted can be narrowed down. Here, as the position of the video object to be extracted, the barycentric coordinates of the video object can be used. As the feature amount, the brightness, color, texture, etc. of the video object can be used.

In the video object trajectory synthesis method, the trajectory image generation step generates a trajectory image in which only the moving trajectory of the video object is drawn from the position and feature amount of the video object extracted in the object extraction step. The trajectory image generated in this manner is sequentially combined with the frame image in the image compositing step, so that a composite video in which the moving trajectory of the video object to be extracted is combined with the video is generated.
The video object trajectory synthesis method includes a search area for searching for the video object in the frame image to be input next based on the position of the video object extracted in the object extraction step by the search area estimation step. The position of the video object to be input next is predicted by applying a Kalman filter to the barycentric coordinates of the object, and is estimated from the prediction.
The search region estimation step updates the Kalman gain, which is the filter sensitivity of the Kalman filter, based on the observation noise covariance that is the covariance of the observation noise and the state covariance that is the covariance of the state quantities in the frame image. The state covariance is small when the tracking of the video object is successful, and is large when the tracking fails.

Furthermore, the video object trajectory synthesis program according to claim 4 extracts and tracks a video object included in the input video, and combines a moving object of the video object with the video to synthesize the moving trajectory of the video object. , And object candidate image generation means, object extraction means, trajectory image generation means, search region estimation means, and image composition means.

According to such a configuration, the video object trajectory synthesis program obtains an object candidate image obtained by extracting video object candidates from frame images in units of still images that are input in time series by the object candidate image generation unit and constitute a video. Generate.
Note that the object candidate image is generated from at least one or more images of a difference image obtained by calculating a difference between a luminance image, a contour image, and frame images having different input times in a frame image input in time series.

Then, the video object locus matching program extracts the position of the video object to be extracted and the feature quantity characterizing the video object from the object candidate images based on the preset extraction condition by the object extracting means.
Here, at least one of the area, luminance, color, aspect ratio, and circularity of the video object to be extracted can be used as the extraction condition. Thereby, even when there are a plurality of video objects in the frame image, the video objects to be extracted can be narrowed down. Here, as the position of the video object to be extracted, the barycentric coordinates of the video object can be used. As the feature amount, the brightness, color, texture, etc. of the video object can be used.

Then, the video object trajectory synthesis program generates a trajectory image in which only the moving trajectory of the video object is drawn from the position and the feature amount of the video object extracted by the object extraction means by the trajectory image generation means. The trajectory image generated in this manner is sequentially combined with the frame image by the image combining means, thereby generating a composite video in which the moving trajectory of the video object to be extracted is combined with the video.
The video object trajectory synthesis program searches the search area for searching for the video object in the next input frame image based on the position of the video object extracted by the object extraction means by the search area estimation means. The position of the video object to be input next is predicted by applying a Kalman filter to the barycentric coordinates of the object, and is estimated from the prediction.
The search area estimation means updates the Kalman gain, which is the filter sensitivity of the Kalman filter, based on the observation noise covariance that is the covariance of the observation noise and the state covariance that is the covariance of the state quantities in the frame image. The state covariance is small when the tracking of the video object is successful, and is large when the tracking fails.

According to the first, third, and fourth aspects of the present invention, it is possible to extract and track a video object to be extracted from a video, and synthesize the moving trajectory of the video object to the video without human intervention. At this time, if a texture is used as the feature amount of the video object, it is possible to generate a composite video in which the motion of the live-action video object is continuously drawn. This makes it possible to provide a video with a new special effect. In addition, it is possible to estimate the position of the moving video object in the frame image, and to limit the search area of the video object based on the position, thereby reducing the amount of calculation when extracting the video object. be able to. This makes it possible to synthesize the moving trajectory of the video object on the video in real time.
Furthermore, according to the present invention, since the composite video in which the motion of the video object is continuously drawn is generated from the video shot by one camera, the scale of the device configuration can be reduced.

  According to the second aspect of the present invention, even when the video object has moved to the invisible region and moved to the visible region again, the movement trajectory can be estimated by estimating the movement of the invisible region.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Configuration of video object trajectory synthesizer]
First, the configuration of the video object trajectory synthesis apparatus according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of a video object trajectory synthesis apparatus.
As shown in FIG. 1, the video object trajectory synthesizer 1 extracts and tracks an object (video object) with a movement to be tracked from an input video, and tracks the movement trajectory of the video object on the video. To be synthesized. Here, the video object locus synthesizing apparatus 1 includes an object candidate image generation means 10, an extraction condition storage means 20, an object extraction means 30, a search area estimation means 40, a locus image generation means 50, and a locus image storage means. 60 and an image composition means 70.

The object candidate image generation means 10 generates an object candidate image obtained by extracting a video object candidate to be tracked for each frame image constituting the video from the input video. The video is composed of, for example, 30 frame images per second. Therefore, the object candidate image generation means 10 extracts a video object candidate from the frame image and binarizes it, thereby generating an image (object candidate image) consisting only of the video object candidate. This object candidate image is an image obtained by extracting a plurality of video objects similar to the video object to be tracked. For example, the object candidate image is an image obtained by roughly extracting a video object to be tracked, such as a video object with movement.
Here, the object candidate image generation means 10 includes a luminance image generation unit 11, a contour image generation unit 12, a difference image generation unit 13, and an object candidate extraction unit 14.

  The luminance image generation unit 11 generates a luminance image obtained by converting a frame image into a monochrome (grayscale) from a frame image of an input video (color video). For example, the luminance image generation unit 11 calculates an average value of the RGB components of the colors constituting each pixel value of the frame image, and generates a luminance image by using the average value as a new pixel value. The luminance image generated here is output to the object candidate extraction unit 14.

  The contour image generation unit 12 generates a contour image in which a contour (edge) is extracted from the frame image of the input video based on the luminance of the frame image. For example, the contour image generation unit 12 detects a contour by detecting a change in luminance of adjacent pixels of the frame image. Note that the contour image generation unit 12 may extract a contour from the luminance image generated by the luminance image generation unit 11. The contour image generated here is output to the object candidate extraction unit 14.

  The difference image generation unit 13 calculates a luminance difference between frame images input at different times (for example, a current frame image and a frame image input before) from frame images input in time series. A difference image as a value is generated. The difference image generation unit 13 may generate a difference image from the luminance images generated at different times by the luminance image generation unit 11. The difference image generated here is output to the object candidate extraction unit 14.

  The object candidate extraction unit 14 is an object candidate image obtained by extracting video object candidates based on the luminance image, the contour image, and the difference image generated by the luminance image generation unit 11, the contour image generation unit 12, and the difference image generation unit 13. Is generated. Here, the object candidate extraction unit 14 includes an image integration unit 141, a binarization unit 142, and a noise removal unit 143.

The image integration unit 141 generates an image (extraction image) for extracting a video object by adding a weight to the luminance image, the contour image, and the difference image. The image for extraction generated here is an image obtained by extracting a plurality of video objects similar to the video object to be tracked from the frame image, and is an image accompanied by the content (texture) of the video object. The extraction image is output to the binarization unit 142.

For example, the image integration unit 141 uses y (x, y, t) as the pixel value of the coordinate (x, y) of the luminance image y at a certain time t, and the pixel value of the coordinate (x, y) as the contour image e. e (x, y, t), the pixel value of the coordinate (x, y) of the difference image d is d (x, y, t), and the respective weighting factors of the luminance image, the contour image, and the difference image are w _y , when a w _e and w _d, coordinates (x, y) of the extracted image g pixel value g of (x, y, t) and is calculated by the following equation (1).

Here the weighting factor (w _y, w _e and w _d) shall be predetermined by the characteristics of the video object to be extracted. For example, when the luminance difference of the video object is larger than that of the background image, the weighting coefficient of the luminance image or the contour image is increased. If the motion of the video object is large, the weighting coefficient of the difference image is increased. In this way, by giving weights to the luminance image, the contour image, and the difference image, the generated image for extraction has a large representation of the pixel value representing the characteristics of the video object to be extracted. become.

  Here, the extraction image is generated from the three images of the luminance image, the contour image, and the difference image. However, when the extraction image is generated from the two images, it is not used from the equation (1). What is necessary is just to delete the term of an image. Further, when only one image is used among the luminance image, the contour image, and the difference image, it is not necessary to provide the image integration unit 141. In this case, the luminance image, the contour image, or the difference image may be input to the binarization unit 142 as it is.

  The binarization unit 142 extracts a plurality of video objects by binarizing the extraction image generated by the image integration unit 141. For example, a binary image is generated by setting the pixel value to “1 (white)” when the pixel value of the extraction image is equal to or greater than a predetermined threshold, and “0 (black)” otherwise. . Thereby, for example, an area where the pixel value is “1 (white)” can be extracted as a video object candidate. The binarized image generated here is output to the noise removing unit 143.

  The noise removing unit 143 removes a fine video object as noise from the binarized image generated by the binarizing unit 142. For example, the noise removing unit 143 removes noise by performing shrinkage processing and expansion processing on the binarized image. Here, the binarized image from which the noise has been removed is output to the object extracting unit 30 as an object candidate image obtained by extracting video object candidates.

  Here, the contraction process is a process for setting a pixel value of a pixel having a pixel value of “0 (black)” to “0 (black)” even in the vicinity of one pixel. The expansion process is a process for setting the pixel value of a pixel having a pixel value of “1 (white)” to “1 (white)” even in the vicinity of one pixel. By this contraction process, a region having a minute pixel value of “1 (white)” can be erased. In this contraction stage, the region having the pixel value “1 (white)” that is the region of the video object is in a contracted state, and thus the region of the video object is expanded by expansion processing. Thereby, a binarized image from which noise is removed is generated.

  In the object candidate extraction unit 14 described above, the binarization unit 142 performs binarization processing on the extraction image generated by the image integration unit 141 and the noise removal unit 143 performs contraction processing and expansion processing. Thus, the binarized image from which noise has been removed is generated, but noise may be removed by performing binarization after smoothing the extraction image.

  The extraction condition storage means 20 stores conditions for selecting a video object to be extracted (tracked), and is a general storage medium such as a hard disk. The extraction condition storage means 20 stores extraction condition information 20a indicating various extraction conditions and position information 20b indicating the position of the video object.

The extraction condition information 20a is information describing the extraction condition of the video object to be extracted, and describes, for example, at least one extraction condition of area, luminance, color, aspect ratio, and circularity. This extraction condition information 20a is a filter (area filter, luminance filter, color filter) for the object extraction means 30 to be described later to select a video object to be extracted from the object candidate images generated by the object candidate image generation means 10. , Aspect ratio filter and circularity filter).
The extraction condition information 20a stores a predetermined initial value and a threshold value indicating an allowable range as conditions for the area filter, the luminance filter, the color filter, the aspect ratio filter, and the circularity filter. As a result, a video object having a value (feature) outside the threshold can be excluded from video object candidates to be extracted.

  Here, the area indicates the number of pixels of the video object, for example. Further, the luminance indicates an average value of the luminance of each pixel in the video object. The color indicates an average value of colors (for example, RGB values) of each pixel in the video object. For this color, the background color of the background image may be set as an initial value, and the degree of change from the background color may be set as a threshold value.

For example, as shown in FIG. 2, in the RGB space, the color (color vector) of the video object is I = [Ir, Ig, Ib], and the background color (color vector) is E = [Er, Eg, Eb]. Then, the change degree α of the brightness (luminance) and the change degree C of the hue (saturation) with respect to the background color can be expressed by the following expressions (2) and (3). Therefore, for example, threshold values θ ₁ and θ ₂ are determined in advance, and a portion where C> θ ₁ or α <θ ₂ is a video object to be extracted.

  The aspect ratio indicates the ratio between the maximum length in the vertical direction and the maximum length in the horizontal direction of the video object. The circularity indicates the degree of circularity of the video object, and has a larger value as it becomes closer to a circular shape. The circularity e is expressed by the following equation (4), where S is the area of the video object and l is the perimeter.

  For example, if the video object to be extracted has a circular shape such as a ball, the circularity of the extraction condition may be set to a value close to 1.

  The position information 20b is information indicating the position of the video object currently being tracked. This position information 20b is, for example, the barycentric position of the video object. The barycentric coordinates are calculated by the feature amount analysis unit 33 of the object extraction unit 30 described later. This position information 20b is an extraction condition for determining the video object closest to the coordinates indicated by the position information 20b as the video object to be extracted when there are a plurality of video objects that match the extraction condition information 20a. Also works.

  The object extraction means 30 is an object to be extracted (tracked) from the object candidate images generated by the object candidate image generation means 10 based on the extraction conditions (extraction condition information 20a) stored in the extraction condition storage means 20. The video object is selected, and the position of the video object and the feature quantity characterizing the video object are extracted. Here, the object extraction unit 30 includes a labeling unit 31, an object selection unit 32, and a feature amount analysis unit 33.

  The labeling unit 31 assigns numbers (labels) to regions that are candidates for video objects in the object candidate images (binarized images) generated by the object candidate image generation unit 10. That is, the labeling unit 31 assigns one number to a connected area (connected area) having a pixel value of “1 (white)” that is an area of the video object. As a result, the video object candidates in the object candidate image are numbered.

  The object selection unit 32 determines, for each video object candidate numbered by the labeling unit 31, whether the video object matches the extraction condition (extraction condition information 20 a) stored in the extraction condition storage unit 20. The video object to be extracted (tracked) is selected. The number (label) of the video object selected here is output to the feature amount analysis unit 33.

  That is, the object selection unit 32 extracts the extraction condition (for example, area, brightness, color, aspect ratio, and circularity) indicated by the extraction condition information 20a stored in the extraction condition storage unit 20 for each video object candidate. Based on the above, by filtering, a video object satisfying the extraction condition is selected as a video object to be extracted.

Further, when performing filtering based on the extraction condition such as the area, the object selection unit 32 updates the extraction condition information 20a with the area of the video object extracted here. Then, when selecting a video object in the next frame image, the extraction condition is determined based on the updated area and the like. As a result, even when the size, brightness, etc. of the video object changes, it is possible to extract the video object in conformity with the extraction condition.
When performing filtering based on color, the object selection unit 32 calculates an average value of colors in the area corresponding to the video object from the frame image of the video input to the video object locus synthesis apparatus 1, and extracts the extraction condition. It is assumed that filtering is performed based on the extraction condition indicated by the information 20a.

  The feature amount analysis unit 33 analyzes the video object corresponding to the label selected by the object selection unit 32 and extracts the feature amount characterizing the position of the video object and the video object. The extracted position and feature amount of the video object are output to the search area estimation unit 40 and the trajectory image creation unit 50. Further, the position of the video object extracted here is written in the position information 20b of the extraction condition storage means 20 as the position information of the current video object. If the object selection unit 32 cannot select a video object that meets the extraction condition, the trajectory image generation unit 50 is notified of this (extraction failure).

  Here, as the position of the video object, the barycentric coordinates of the video object, the vertex coordinates of the polygon approximation, the control point coordinates of the spline curve, and the like can be used. When a plurality of video objects are selected by the object selection unit 32, for example, the video object closest to the position of the video object extracted from the frame image one frame before is specified as the video object being tracked.

  In addition, a video object existence area, luminance, color, texture, and the like can be used as the feature amount of the video object. The existence area is used by the search area estimation means 40 to estimate the search area of the video object in the next frame image. For the luminance and color, the values used when the object selection unit 32 compares with the extraction condition can be used. The texture is used when a trajectory image is generated by the trajectory image generation means 50 described later using an image obtained by extracting an area corresponding to the video object from the video frame image input to the video object trajectory synthesis apparatus 1. be able to.

The search area estimation means 40 estimates the search area of the video object in the next input frame image based on the position (center of gravity coordinates, etc.) of the video object extracted by the object extraction means 30.
This search area estimation means 40 predicts the position of the video object in the next frame image, for example, by applying a Kalman filter to the barycentric coordinates, and estimates the search area. Hereinafter, a method in which the search region estimation unit 40 estimates the search region using the Kalman filter will be described. The Kalman filter changes from moment to moment by applying a recurrence formula that performs “filtering” to estimate the current state based on the observed quantity observed in time series and “prediction” to estimate the future state. The state to be performed is estimated.

Here, it is assumed that the search area estimation unit 40 estimates the state quantity x shown in the following equation (5) including the barycentric coordinates (p _x , p _y ) of the video object. Note that v _x and v _y are the velocities of the video object in the horizontal direction (x direction) and the vertical direction (y direction), respectively. T represents transposition of the matrix.

First, the search area estimation means 40 has an observation noise covariance R that is a covariance of observation noise and a state covariance P _t− that is a covariance of a state quantity x _t−1 in a frame image at time (t−1). based on ₁ and updates the showing filter sensitivity (Kalman gain) K below (6).

Here, the matrix H is an observation matrix indicating the relationship between the state quantity x and the observation quantity y that is the barycentric coordinate input from the object extraction means 30. T represents transposition of the matrix.
Note that the observation matrix H is a matrix represented by the following formula (7) as a matrix for observing the barycentric coordinates (p _x , p _y ) of the state quantity x.

Further, observation noise covariance R is the horizontal direction r _x observation accuracy of the video object, when the vertical direction r _y, and covariance shown in the following equation (8).

  Furthermore, the search area estimation means 40 updates the state quantity x and the state covariance P according to the following expressions (9) and (10).

Here, K is the filter sensitivity (Kalman gain) calculated by equation (6), y is the barycentric coordinates (observation amount) input from the object extraction means 30, and H is the observation matrix of equation (7). Thus, the search area estimation means 40 estimates the current state based on the equations (6), (9), and (10) (state update equation).
Additionally, the search region estimating means 40 based on the state amount x _t and state covariance P _t at time t of the frame image, the time the state quantity in the frame image of the _{(t + 1) x t +} 1 and the state covariance P _{t + 1,} predicted by shown below (11) and (12).

  Here, F is a state transition rule set in advance and represents a state transition matrix. In this case, it is assumed that the video object has a constant linear motion, and the state transition matrix is a matrix shown in the following equation (13). T represents transposition of the matrix.

Further, Q represents system noise covariance (process covariance) set in consideration of modeling errors of constant velocity linear motion and disturbances such as noise. For example, when errors occur independently with respect to the horizontal and vertical velocities v _x and v _{y of} the video object, the process covariance Q can use a matrix represented by the following equation (14). Note that q _x and q _y are non-negative values.

As described above, the search area estimation unit 40 can estimate the barycentric coordinates of the video object in the next frame image based on the equations (11) and (12) (state prediction equation).
When applying the Kalman filter, the state covariance may be controlled according to the tracking state. For example, when the tracking of the video object is successful, the state covariance may be reduced, and when the tracking fails, the state covariance may be increased.

In addition, the search area estimation means 40 can change the search area of the video object in the frame image in association with the change in the state covariance. For example, the search area estimation means 40 estimates the search area of the minimum size by adding the state covariance to the existence area that is the feature amount of the video object input from the object extraction means 30. Can do.
The description of the configuration of the video object locus synthesizing device 1 in FIG. 1 will be continued.

  The trajectory image generation means 50 draws an image indicating the feature amount of the video object based on the position and feature amount of the video object extracted by the object extraction means 30. Here, the trajectory image generating means 50 includes a feature amount image drawing unit 51 and a trajectory interpolation unit 51.

  The feature amount image drawing unit 51 draws a feature amount at a location corresponding to the position of the video object extracted by the object extraction unit 30 on the locus image 60a stored in the locus image storage unit 60. . For example, when a color is used as the feature amount, the color is drawn on the trajectory image 60a with a predetermined size (for example, the size of a ball).

  Further, for example, when a texture is used as the feature amount, the video object on the frame image can be drawn as it is by drawing the texture on the trajectory image 60a. The feature amount image drawing unit 51 draws the trajectory image 60a every time the object extraction unit 30 is notified of the position and feature amount of the video object. As a result, an image representing the locus of only the video object being tracked is generated. Also, the feature image drawing unit 51 instructs the image composition unit 70 to perform image composition at the timing when the trajectory image 60a is created.

  Further, the feature amount image drawing unit 51 stores at least the latest coordinates where the feature amount is drawn in a storage unit (not shown). The coordinates (latest drawing position) are used to interpolate the trajectory in the trajectory interpolation unit 52 described later when the object extraction unit 30 fails to extract the video object. It should be noted that the feature amount image drawing unit 51, after failing to extract the video object, at the stage of successful extraction, the trajectory interpolation unit 52, with respect to the current video object position, the latest drawing position, and the video object The feature amount (for example, texture) is output to the trajectory interpolation unit 52.

  The trajectory interpolation unit 52, based on the current video object position, the latest drawing position, and the video object feature amount, which are output from the feature amount image drawing unit 51, in the trajectory image 60a, The video object presumed to exist between the positions of the video objects is interpolated by interpolation. As a result, for example, when a so-called occlusion occurs that hides a video object that exists behind the video object that exists in the foreground, the position of the video object is interpolated even if the video object that exists behind is the tracking target. The locus image 60a can be drawn.

  The trajectory image storage means 60 stores the trajectory image generated by the trajectory image generation means 50 and is a general storage medium such as a hard disk. The trajectory image 60a stored in the trajectory image storage unit 60 is generated by the feature amount image drawing unit 51 of the trajectory image generation unit 50 in which the feature amount (for example, texture) of the video object is drawn at the timing of the frame image input interval. Is done. Alternatively, the locus interpolation unit 52 interpolates the position of the video object while the video object extraction has failed. Further, the locus image 60 a drawn here is read by the image composition means 70.

  The image synthesizing unit 70 synthesizes the trajectory image 60a stored in the trajectory image storage unit 60 for each frame image constituting the video, thereby combining the video with the trajectory of the video object to be tracked. Is to be generated. The image synthesizing unit 70 synthesizes the frame image and the trajectory image 60 a based on the image synthesis instruction notified from the trajectory image generating unit 50.

  The configuration of the video object trajectory synthesizing device 1 has been described based on one embodiment, but the present invention is not limited to this. For example, when the image composition means 70 synthesizes the feature quantity of the video object with the frame image, in the case of the feature quantity such as color and brightness, it is not necessary to store the trajectory image 60a, and the video object has moved. By storing only the position coordinates, the trajectory of the video object may be drawn and combined with the position coordinates based on the feature amount such as the color and the luminance when the image is combined.

  The video object locus synthesizing apparatus 1 described above can be realized by causing a general computer to execute a program and operating an arithmetic device or a storage device in the computer. This program (video object trajectory synthesis program) can be distributed via a communication line, or can be distributed on a recording medium such as a CD-ROM.

(Example of video object trajectory synthesis)
Here, with reference to FIG. 3 and FIG. 4, an example in which the video object trajectory synthesis apparatus 1 synthesizes the trajectory of the video object will be described. FIG. 3 is an explanatory diagram for explaining an example in which the moving trajectory of the video object is synthesized on the video. FIG. 4 is an explanatory diagram for explaining an example of interpolating a movement trajectory when extraction of a video object fails. In the following description, it is assumed that a baseball is tracked and the movement trajectory is synthesized.

  First, referring to FIG. 3 (refer to FIG. 1 as appropriate), an example of synthesizing the moving trajectory of the video object on the video will be described. FIG. 3A shows the frame images constituting the video input to the video object locus synthesizing device 1 in time series. FIG. 3B shows, in time series, the trajectory image of the video object generated by the trajectory image generation means 50 corresponding to each frame image of FIG. FIG. 3C shows, in chronological order, a combined frame image obtained by combining the frame image and the trajectory image by the image combining unit 70 corresponding to each frame image of FIG.

As shown in FIG. 3A, the video object trajectory synthesis device 1 sequentially extracts balls M ₁ , M ₂ ... M _n from the frame images at time t, time (t + 1)... Time (t + n). Then, as shown in FIG. 3B, a trajectory image in which a ball (here, texture) extracted from the frame image of FIG. 3A is generated. Here, as shown in the trajectory image at time (t + 1) in FIG. 3B, the trajectory image is obtained by adding the current time (time) to the trajectory image created by the ball M ₁ generated at the previous time (time t). The ball M ₂ extracted in (t + 1)) is drawn. As a result, only the ball (texture) is drawn as a continuous trajectory in the trajectory image.

  Then, the video object locus synthesizing apparatus 1 synthesizes the frame image (FIG. 3 (a)) and the locus image (FIG. 3 (b)) at the same time, so that time t, time (t + 1)... Time (t + n) ) Each time a composite frame image (FIG. 3C) is generated. In addition, the composite frame image is output in time series, thereby forming a composite video. Thereby, an image (composite image) in which the trajectory of the ball is drawn can be generated.

  Next, with reference to FIG. 4 (refer to FIG. 1 as appropriate), an example of interpolating a movement trajectory when extraction of a video object fails will be described. FIG. 4A shows the frame images constituting the video input to the video object locus synthesizing device 1 in time series. FIG. 4B shows, in time series, the trajectory image of the video object generated by the trajectory image generation means 50 corresponding to each frame image of FIG. FIG. 4C shows, in chronological order, a synthesized frame image obtained by synthesizing the frame image and the trajectory image by the image synthesizing unit 70 corresponding to each frame image of FIG.

As shown in FIG. 4A, the video object trajectory synthesizing device 1 sequentially extracts balls M ₁ ... M ₃ and M ₄ from frame images at time t... Time (t + 2) and time (t + 3). Then, as shown in FIG. 4B, a trajectory image in which a ball (texture) extracted from the frame image of FIG.
Here, it is assumed that the extraction of the ball failed at time (t + 1) (not shown) in FIG. 4A and the time (t + 2) shown in FIG. 4A, and the extraction of the ball was successful at time (t + 3). . At this time, the ball that failed to be extracted is not drawn in the trajectory image at time (t + 2) in FIG. That is only the M ₁ was successfully extracted is drawing state. Therefore, only the ball M ₁ is synthesized with the synthesized frame image at time (t + 2) in FIG. 4C (where M ₃ is the ball of the frame image).

At time (t + 3), when the ball is successfully extracted, the trajectory interpolation unit 52 of the trajectory image generation means 50 draws the ball M ₄ on the trajectory image, and the ball M ₁ and the ball M on the trajectory image. Balls M ₂ ′ and M ₃ ′ are drawn at positions calculated by interpolation from the position of ₄ (for example, the position of the center of gravity). At this time, the same feature amount as that of the ball M ₄ is used as the feature amount. As described above, even when the extraction of the video object fails, the ball trajectory continues as shown in the composite frame image at time (t + 3) in FIG. A video (composite video) drawn can be generated.

[Operation of video object trajectory synthesizer]
Next, the operation of the video object locus synthesizing apparatus 1 will be described with reference to FIGS. FIG. 5 is a flowchart showing the overall operation of the video object trajectory synthesis apparatus 1. FIG. 6 is a flowchart showing the operation of the video object selection process in the object selection unit 32 of the object extraction means 30. FIG. 7 is a flowchart showing the operation of the trajectory image drawing process in the trajectory image generation means 50.
First, the operation of the video object trajectory synthesis apparatus 1 will be described with reference to FIG.

(Object candidate image generation step)
The video object locus synthesizing apparatus 1 sets a search area for searching for a video object in the entire range of the frame image in the first frame image constituting the input video by the object candidate image generation means 10 (step S10).
Further, the video object trajectory synthesis apparatus 1 uses the luminance image generation unit 11 to generate a luminance image obtained by converting the search area in the frame image into monochrome (grayscale). In the video object trajectory synthesis device 1, the contour image generation unit 12 generates a contour image in which a contour (edge) is extracted from the search area in the frame image by luminance. Furthermore, the video object locus synthesizing apparatus 1 uses the difference image generation unit 13 to generate a difference image in which the luminance difference between the frame images input at different times is a pixel value in the search area (step S11).

Then, the video object trajectory synthesis device 1 integrates the luminance image, the contour image, and the difference image generated in step S11 by the image integration unit 141 of the object candidate extraction unit 14 by applying a predetermined weighting coefficient (see above). Thus, an image (extraction image) for extracting a video object is generated (see formula (1)) (step S12).
Further, the video object trajectory synthesis device 1 binarizes the extraction image generated in step S12 by the binarization unit 142, and contracts and expands the binarized image by the noise removal unit 143. By performing processing, an image from which noise has been removed is generated. As a result, an object candidate image obtained by extracting video object candidates is generated within the search area of the frame image (step S13).

(Object extraction step)
Then, the video object locus synthesizing apparatus 1 uses the labeling unit 31 of the object extraction unit 30 to assign a number (label) to a region that is a candidate for the video object in the object candidate image (step S14). The subsequent operations are processed in units of video objects based on the numbers assigned to the video object candidates.

  Further, the video object trajectory synthesis device 1 uses the object selection unit 32 to extract the extraction condition (extraction condition) stored in the extraction condition storage unit 20 from the object candidate images for each video object numbered in step S14. Based on the information 20a), a video object to be extracted (tracked) is selected (step S15). The detailed operation of the video object selection process in step S15 will be described later with reference to FIG.

  Then, the video object trajectory synthesis device 1 analyzes (calculates) the feature quantity of the video object selected in step S15 by the feature quantity analysis unit 33, and extracts the position and feature quantity of the video object (step S16). . As the position of the video object, for example, the barycentric coordinates of the video object are used. In addition, as the feature amount of the video object, the existence area, luminance, color, texture, and the like of the video object are used.

(Track image generation step)
Further, the video object trajectory synthesis apparatus 1 draws an image indicating the feature amount of the video object by the feature amount image drawing unit 51 of the trajectory image generation unit 50 and stores the image as the trajectory image 60a in the trajectory image storage unit 60 (step). S17). In step S17, a trajectory image 60a is generated by drawing a feature amount (eg, texture) at the position (eg, barycentric coordinates) of the video object extracted in step S16. The detailed operation of the trajectory image drawing process in step S17 will be described later with reference to FIG.

(Image composition step)
Then, the video object trajectory synthesis apparatus 1 uses the image synthesis means 70 to synthesize the trajectory image 60a stored in the trajectory image storage means 60 and the frame image constituting the input video to generate a composite frame image. (Step S18). The image composition means 70 continuously outputs the composite frame image, so that a composite video in which the moving trajectory of the video object is combined with the video is generated.
Further, the video object locus synthesizing apparatus 1 estimates the search area for the video object in the next input frame image based on the position (center of gravity coordinates, etc.) of the video object extracted in step S16 (step S19). Note that a Kalman filter can be used for estimation of the search region.

  Here, the video object locus synthesizing apparatus 1 determines whether or not there is a frame image (next frame image) to be processed next. If there is a next frame image (step S20; Yes), the process proceeds to step S11. Returning, the video object is extracted in the search area estimated in step S19. On the other hand, when the next frame image does not exist (step S20; No), the operation ends.

  Through the above operation, the video object trajectory synthesis apparatus 1 sequentially extracts and tracks video objects to be tracked from frame images input in time series as video, and combines the video object movement trajectory with video. Can be output. Further, since the video object trajectory synthesis apparatus 1 extracts the video object while appropriately updating the area for searching for the video object, it is possible to reduce the amount of calculation for extracting and updating the video object.

[Video object selection processing operation]
Next, referring to FIG. 6 (refer to FIG. 1 as appropriate), the operation of the video object selection process (step S15 in FIG. 5) in the object selection unit 32 of the video object locus synthesis apparatus 1 will be described.
First, the object selection unit 32 selects one video object from the object candidate images based on the number (label) (step S30).

  Then, the object selection unit 32 determines whether or not the “area” of the selected video object matches the extraction condition (extraction condition information 20a) stored in the extraction condition storage unit 20 (step S31). Here, when the matching condition by the “area” is met (step S31; condition matching), the object selection unit 32 determines whether or not the “color” of the video object matches the extraction condition (step S32). Further, when the matching condition based on “color” is met (step S32; condition matching), the object selection unit 32 determines whether or not the “luminance” of the video object matches the extraction condition (step S33). If the matching condition by “luminance” is met (step S33; condition matching), the object selection unit 32 determines whether the “aspect ratio” of the video object matches the extraction condition (step S34). Further, when the matching condition based on the “aspect ratio” is met (step S34; condition matching), the object selection unit 32 determines whether the “circularity” of the video object matches the extraction condition (step S35). .

  Furthermore, when the conformity condition based on the “circularity” is met (step S35; condition conformance), that is, when the extraction condition is met in all of steps S31 to S35, the image object selected in step S30 is selected as the image to be tracked. It selects as an object (step S36) and progresses to step S37. On the other hand, in any of steps S31 to S35, if the extraction condition is not met (condition non-conformity), the process proceeds to step S37 as it is.

Then, the object selection unit 32 determines whether or not the conformity determination has been performed on all the video objects of the object candidate image (step S37), and when the conformance determination has been performed on all the video objects (Yes), the operation ends. To do. On the other hand, if there is a video object that has not yet been determined for suitability (No), the process returns to step S30 to continue the operation.
With the above operation, the video object trajectory synthesis apparatus 1 can select a video object suitable for the extraction condition from the object candidate images by the object selection unit 32.

[Video object selection processing operation]
Next, referring to FIG. 7 (refer to FIG. 1 as appropriate), the operation of the trajectory image drawing process (step S17 in FIG. 5) in the trajectory image generation means 50 of the video object trajectory synthesis apparatus 1 will be described.
First, the trajectory image generation means 50 determines whether or not the object extraction means 30 has succeeded in extracting a video object (step S40). If the extraction of the video object has failed (step S40; No), the operation ends. On the other hand, if the video object has been successfully extracted (step S40; Yes), it is determined whether the video object has been successfully extracted from the previous frame image (step S41).

If the video object has been successfully extracted from the previous frame image (step S41; Yes), the process proceeds to step S43. On the other hand, if the extraction of the video object has failed in the previous frame image, that is, if the feature amount (for example, texture) of the video object before the previous frame image has not been drawn in the trajectory image 60a, the trajectory image generation The trajectory interpolation unit 52 of the means 50 interpolates the position of the current video object and the position of the video object that has been successfully drawn by interpolation, and adds the texture of the current video object to the interpolated position. Drawing is performed (step S42).
Then, the feature amount image drawing unit 51 of the locus image generation means 50 draws the locus image 60a based on the feature amount (texture or the like) at the current position of the video object based on the feature amount of the video object (step S43). ).

  With the above operation, the video object trajectory synthesis apparatus 1 generates a trajectory image by interpolation even when video object extraction fails in the middle. The movement trajectory of the selected video object is synthesized.

It is the block diagram which showed the structure of the video object locus | trajectory synthesis apparatus based on this invention. It is explanatory drawing for demonstrating the method of extracting a video object with a color. It is explanatory drawing for demonstrating the example which synthesize | combines the movement locus | trajectory of a video object on a video. It is explanatory drawing for demonstrating the example which interpolates a movement locus | trajectory when extraction of a video object fails. It is a flowchart which shows the whole operation | movement of the video object locus | trajectory synthesis apparatus which concerns on this invention. It is a flowchart which shows the operation | movement of the selection process of the video object in the video object locus | trajectory synthesis apparatus which concerns on this invention. It is a flowchart which shows the operation | movement of the drawing process of the locus | trajectory image in the video object locus | trajectory synthesis apparatus which concerns on this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image | video object locus | trajectory synthesis apparatus 10 Object candidate image generation means 20 Extraction condition memory | storage means 30 Object extraction means 40 Search area estimation means 50 Trajectory image generation means 60 Trajectory image storage means 70 Image composition means

Claims (4)

A video object trajectory synthesizer that extracts and tracks a video object included in an input video and synthesizes a moving trajectory of the video object on the video,
Object candidate image generation means for generating an object candidate image obtained by extracting the video object candidates from time-series input frame images constituting the video;
Extraction condition storage means for storing extraction conditions for the video object to be extracted;
An object for extracting the position of the video object and the feature quantity characterizing the video object from the object candidate images generated by the object candidate image generation unit based on the extraction conditions stored in the extraction condition storage unit Extraction means;
Based on the position and feature amount of the video object extracted by the object extraction means, an image indicating the feature amount of the video object is drawn in a corresponding area of the frame image, thereby extracting the movement trajectory of the video object. A trajectory image generating means for generating a trajectory image obtained;
Image synthesizing means for synthesizing the trajectory image generated by the trajectory image generating means and the frame image;
By applying a Kalman filter to the barycentric coordinates of the object, a search area for searching the video object in the next input frame image based on the position of the video object extracted by the object extraction means Search area estimation means for predicting the position of the video object to be input next and estimating from the prediction;
The search region estimation means, based on the Kalman gain that is the filter sensitivity of the Kalman filter, based on the observation noise covariance that is the covariance of the observation noise and the state covariance that is the covariance of the state quantities in the frame image, update, the state covariance, the small when successful tracking of video objects, video object locus synthesis device according to claim Rukoto greatly when it fails to track.   The trajectory image generation means includes trajectory interpolation means for interpolating the position of the video object that has failed to be extracted from the position of the video object that has been successfully extracted based on the extraction result of the object extraction means. The video object locus synthesizing device according to claim 1. A video object trajectory synthesis method that extracts and tracks a video object included in an input video and synthesizes a moving trajectory of the video object on the video,
Extracting candidates of the video object from at least one of a difference image obtained by calculating a difference between a luminance image, a contour image, and a frame image having different input times in a frame image that is configured and configured in time series. An object candidate image generation step for generating an object candidate image that has been performed;
Features that characterize the position of the video object and the video object based on at least one extraction condition of area, brightness, color, aspect ratio, and circularity from the object candidate image generated in the object candidate image generation step An object extraction step to extract the quantity;
Based on the position and feature amount of the video object extracted in the object extraction step, an image indicating the feature amount of the video object is drawn in a corresponding area of the frame image, thereby extracting the movement trajectory of the video object. A trajectory image generation step for generating a trajectory image,
An image synthesis step for synthesizing the trajectory image generated in the trajectory image generation step and the frame image;
Based on the position of the video object extracted in the object extraction step, a search area for searching for the video object in the frame image to be input next is applied by applying a Kalman filter to the barycentric coordinates of the object. A search region estimation step of predicting the position of the video object input to
The search region estimation step is based on Kalman gain, which is filter sensitivity of the Kalman filter, based on observation noise covariance that is covariance of observation noise and state covariance that is covariance of state quantities in the frame image. updating the state covariance, the small when successful tracking of video objects, video object locus synthesis method and characterized make a marked when it fails to track. In order to extract and track a video object included in the input video, and to synthesize a moving locus of the video object on the video,
Extracting candidates of the video object from at least one of a difference image obtained by calculating a difference between a luminance image, a contour image, and a frame image having different input times in a frame image that is configured and configured in time series. Object candidate image generation means for generating the object candidate image
Features that characterize the position of the video object and the video object based on at least one extraction condition of area, brightness, color, aspect ratio, and circularity from the object candidate image generated by the object candidate image generation unit Object extraction means for extracting the quantity,
Based on the position and feature amount of the video object extracted by the object extraction means, an image indicating the feature amount of the video object is drawn in a corresponding area of the frame image, thereby extracting the movement trajectory of the video object. A trajectory image generating means for generating a trajectory image obtained;
Image synthesis means for synthesizing the trajectory image generated by the trajectory image generation means and the frame image;
Based on the position of the video object extracted by the object extraction means, a search area for searching for the video object in the next input frame image is applied, and a Kalman filter is applied to the barycentric coordinates of the object. Function as search area estimation means for predicting the position of the video object input to
The search region estimation means, based on the Kalman gain that is the filter sensitivity of the Kalman filter, based on the observation noise covariance that is the covariance of the observation noise and the state covariance that is the covariance of the state quantities in the frame image, update, the state covariance, the small when successful tracking of video objects, video object locus synthesis program characterized Rukoto greatly when it fails to track.

Images