JP4441354B2 - Video object extraction device, video object trajectory synthesis device, method and program thereof - Google Patents

Description

  The present invention relates to a video object extraction apparatus, a video object trajectory synthesis apparatus, a method thereof, and a program thereof that extract a video object included in an input video and synthesize a moving trajectory of the video object on the video.

Conventionally, there are various video object extraction techniques for extracting and tracking an object (video object) included in a video (see, for example, Patent Document 1).
Attempts have been made to expand the scope of video production by using the positional information of the extracted video objects to give the video special effects. As an example of giving a special effect to the video, the ball (video object) thrown by the pitcher is tracked in a baseball broadcast, etc., and then the movement trajectory of the ball is displayed on the pitcher's pitch video by CG (computer graphics). There is something to synthesize. Thereby, the viewer can more feel the change of the trajectory of the ball, for example, the change ball, from the moment when the pitcher releases the ball until the catcher catches the ball.

The video object tracking device disclosed in Patent Document 1 generates a silhouette image obtained by binarizing an input video signal into a foreground image and a background image, and estimates motion prediction of the video object from the video object areas of the past two frames. Is. This conventional video object tracking device extracts and tracks a video object from a video by performing a luminance difference extraction process from two continuous images, and processes a video once stored. That is, the moving trajectory of the video object is not synthesized in real time based on the extracted position information of the video object.
Therefore, the applicant of the present application has applied for a video object trajectory synthesis device (Japanese Patent Application No. 2003-355620) that synthesizes a moving trajectory of a video object in real time. In this video object trajectory synthesis apparatus, when tracking a moving video object, it is necessary to predict the position coordinates in the next image of the video object, and to set the search area around the predicted point. As such a position prediction method, the video object trajectory synthesizer previously applied for linear prediction is currently used.
As an example of linear prediction, there is a method of calculating a motion vector using block matching or optical flow (see, for example, Patent Document 2).
JP 2003-44860 (paragraphs 0018 to 0027, FIG. 1) JP 2002-150291 A (paragraphs 0050 to 0060, FIG. 3)

  However, in the conventional video object extraction technique, since luminance difference extraction processing is performed from two continuous images, a lot of noise other than the target video object is generated. As an example, in baseball broadcasts, when there are other moving objects around a video object (ball) to be extracted, there are many cases of erroneous extraction. Specifically, the hands of the pitcher and the catcher are extracted in addition to the ball, and even these afterimages are displayed in the synthesized video. In addition, since the video facing the batter from the pitcher in the baseball broadcast is a video from the right direction due to the camera arrangement, when the left batter enters the batter box, the left batter exists on the ball trajectory. Since this left batter acts as a large animal body compared to the ball, a large amount of noise components are generated in the drawn image. For this reason, ball extraction often fails. Therefore, for the left batter, a composite image (ball trajectory display) in which the ball trajectory is combined with the video is hardly used. Further, since the trajectory of the ball thrown by the pitcher is not a straight line, the predicted ball position near the batter cannot be obtained accurately only by linear prediction. For this reason, it is necessary to take a wide search area for the ball, and there is a problem in that the amount of calculation for extracting the ball cannot be suppressed.

  The present invention has been made in view of the above problems, and a video object extraction apparatus and a video object trajectory synthesis capable of stably extracting and tracking a video object even when noise exists in a background image. An object is to provide an apparatus, a method thereof, and a program thereof.

  The present invention has been made to achieve the above object, and first, the video object extraction device according to claim 1 is a video object extraction device for extracting the position of a video object included in an input video. And an object candidate image generation means and an object extraction means, wherein the object candidate image generation means includes a first image storage means, a second image storage means, and a first difference image generation means. And a second difference image generation means and a candidate image generation means.

  According to such a configuration, the video object extraction device includes the first image (past), the second image (current), and the third image (future) that are input in time series by the object candidate image generation unit. Among these, the first image is stored in the first image storage means, and the second image is stored in the second image storage means. Further, the object candidate image generation means subtracts the brightness of the newly input third image from the brightness of the second image stored in the second image storage means, for example, by the first difference image generation means. Thus, the first difference image is generated. Further, the second difference image generation means subtracts the brightness of the second image stored in the second storage means from the brightness of the first image stored in the first storage means, thereby generating the second difference image generation means. A difference image is generated. These first difference image and second difference image are input to the candidate image generating means. The candidate image generating means refers to the pixel value arranged at the common position of the first difference image and the second difference image, the sign of the pixel value of the first difference image, and the pixel value of the second difference image An area with a different code is discriminated, and an object candidate image is generated.

  Here, if the video object in the input image moves at a high speed and the brightness is higher than that of the background image, the candidate image generating means has a plus sign of the pixel value of the first difference image and the first image. A region where the sign of the pixel value of the difference image 2 is minus is extracted as a video object candidate image. That is, the video object candidate image is an image shown in a region where the sign of the pixel value of the first difference image is different from the sign of the pixel value of the second difference image. Also, when the video object in the input image has a lower brightness than the background image, or when the method of generating the difference image is changed, the sign of the pixel value of the first difference image and the pixel of the second difference image A region having a different value sign can be set as a video object candidate image.

  Then, in the video object extraction device, the object extraction unit refers to the extraction condition, and extracts the position of the video object to be extracted from the object candidate images. For example, it is possible to specify one video object from a plurality of video object candidates by storing the extraction conditions of the video object to be extracted in advance in the storage means. Thereby, the position in the image of the video object to be extracted is specified. Here, the position of the video object can use the barycentric coordinates of the video object.

  3. The video object extracting device according to claim 2, wherein the video object in the next input image is based on the position of the video object extracted by the object extracting means in the video object extracting device according to claim 1. An object position predicting means for predicting the position of the video object and outputting the predicted position information of the video object to the object candidate image generating means.

  According to such a configuration, the video object extraction device uses the object position prediction unit to search the video object search area in the next input field image based on the position of the video object extracted by the object extraction unit. Set in the neighborhood area of the object.

  The video object extracting device according to claim 3 is the video object extracting device according to claim 2, wherein the object position predicting means predicts that the trajectory of the video object is a straight line; Curve prediction means for predicting that the locus becomes a curve, and linear prediction by the linear prediction means and curve prediction by the curve prediction means based on the number of position coordinates of the video object extracted in the past or the equation of the curve And switching means for switching between.

  According to such a configuration, the object position prediction means performs linear prediction when, for example, the position coordinates extracted in the past of the video object are smaller than a predetermined number, and curves prediction when the number is larger than the predetermined number. Execute. For example, when the curve prediction is executed and the curve equation is updated beyond the allowable range, the prediction is switched to the straight line prediction. Moreover, the linear prediction means can estimate the position of the video object which moves sequentially by using a Kalman filter, for example, and can limit the search area. Further, the curve predicting means predicts a curve as a prediction trajectory of the video object by, for example, the least square method.

  According to a fourth aspect of the present invention, there is provided a video object trajectory synthesis device based on the video object extraction device according to any one of the first to third aspects and a position of the video object extracted by the object extraction means. A drawing means for generating a trajectory image obtained by extracting a moving trajectory of the video object by drawing an image showing the video object in an area corresponding to the image inputted in the series; and the trajectory image and the time series input And an image synthesizing means for synthesizing the image to be synthesized.

  According to such a configuration, the video object trajectory synthesis apparatus generates a trajectory image in which only the moving trajectory of the video object is drawn from the position of the video object extracted by the object extraction means by the drawing means. The trajectory image can be generated by sequentially overwriting the video objects extracted for each input image in time series. The trajectory image generated in this manner is sequentially combined with the input image by the image compositing means, thereby generating a composite video in which the moving trajectory of the video object to be extracted is combined with the video.

  The video object trajectory synthesis method according to claim 5 includes a difference image generation step, an object candidate image generation step, an object extraction step, a drawing step, and an image synthesis step.

  According to this procedure, the video object trajectory synthesis method generates the first difference image and the second difference image from the continuous images input in time series by the difference image generation step. Then, in the object candidate image generation step, an object candidate image is generated by extracting an area where the code of the pixel value of the first difference image is different from the code of the pixel value of the second difference image as a video object candidate. In the object extraction step, the position of the video object to be extracted is extracted from the object candidate images. Here, at least one of the position, area, luminance, color, and circularity of the video object to be extracted is used as the extraction condition. Then, the video object trajectory synthesis method generates a trajectory image in which only the moving trajectory of the video object is drawn in the drawing step. The trajectory image is sequentially combined with the input image in an image compositing step, thereby generating a composite video in which the moving trajectory of the video object to be extracted is combined with the video.

  Further, the video object locus synthesis program according to claim 6 causes the computer to function as a difference image generation means, an object candidate image generation means, an object extraction means, a drawing means, and an image composition means.

  According to such a configuration, the video object trajectory synthesis program generates the first difference image and the second difference image from the continuous images input in time series by the difference image generation means. Then, an object candidate image is generated by extracting an area where the code of the pixel value of the first difference image is different from the code of the pixel value of the second difference image by the object candidate image generating means. Then, the position of the video object to be extracted is extracted from the object candidate image by the object extracting means. Further, the video object trajectory synthesis program generates a trajectory image in which only the moving trajectory of the video object is drawn by the drawing means. The trajectory image is sequentially synthesized with the field image by the image synthesizing means, thereby generating a synthesized video in which the moving trajectory of the video object to be extracted is synthesized with the video.

  According to the first aspect of the present invention, the object candidate image generating means extracts the region where the pixel values of the two difference images have different signs as the object candidate image. Therefore, in the vicinity of the video object to be extracted, Even if other moving objects as noise exist, it is possible to stably extract only video objects that move at high speed.

  According to the second aspect of the present invention, it is possible to estimate the position of the video object that moves sequentially in the input image, and to set the search area of the video object in the vicinity of the video object based on the position. Therefore, it is possible to reduce the amount of computation when extracting the video object.

  According to the third aspect of the invention, since linear prediction and curve prediction are switched, position prediction with high accuracy and reduced false detection can be performed, and a video object can be tracked stably.

  According to the invention of claim 4, claim 5 or claim 6, the video object to be extracted is extracted and tracked from the video, and the movement trajectory of the video object is synthesized with the video without human intervention. Can do. This makes it possible to provide a video with a new special effect.

Next, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.
First, the configuration of the video object trajectory synthesis apparatus according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of a video object trajectory synthesis apparatus.
As shown in FIG. 1, the video object trajectory synthesizer 1 extracts and tracks an object (video object) with a movement to be tracked from an input video, and tracks the movement trajectory of the video object on the video. To be synthesized. Note that the video object to be tracked is premised on a ball, for example. Here, the video object locus synthesizing apparatus 1 includes a video object extracting apparatus 10, a video delay unit 20, and a drawing / image synthesis unit 30.

The video object extraction device 10 extracts the position coordinates and feature quantities of the ball from the input video, and outputs the extracted position coordinates and feature quantities to the drawing / image synthesis means 30. The video object extraction device 10 may be configured to extract and output only the position coordinates.
The video delay means 20 delays the input video and outputs it to the drawing / image synthesis means 30.
The drawing / image synthesizing means 30 includes a drawing means 31 for drawing a ball trajectory by CG (computer graphics) at a place corresponding to the position of the ball extracted by the video object extracting device 10, and a trajectory of the drawn ball. An image synthesizing unit 32 that synthesizes the image and the video delayed by the video delay unit 20 and outputs a synthesized video is provided. The drawing / image composition means 30 includes an interpolation means (not shown) for interpolating the trajectory of the ball by interpolation when the video object extraction device 10 fails to extract the ball.

As shown in FIG. 1, the video object extraction device 10 includes an object candidate image generation unit 11, a ball selection unit 12, an extraction condition storage unit 13, a position prediction unit 14, and a search area setting unit 15. Yes.
The object candidate image generation means 11 generates an object candidate image by extracting a search area for each field image constituting the video from the input video and extracting video object candidates to be tracked. The video is composed of, for example, 60 field images per second. Therefore, the object candidate image generation means 11 extracts a video object candidate from the field image and binarizes it, thereby generating an image (object candidate image) consisting only of the video object candidate. . This object candidate image is an image obtained by extracting a plurality of video objects similar to the video object to be tracked. For example, the object candidate image is an image obtained by roughly extracting a video object to be tracked, such as a video object with movement.

The object candidate image generation unit 11 includes image storage units 111 and 112, difference image generation units 113 and 114, and candidate image generation unit 115.
The image storage units 111 and 112 are memories for recording, for example, video signals as digital data in units of one field and performing various signal / image processing.
The image storage unit 111 (second image storage unit) stores an input image (odd field and even field) at an intermediate position (current) among three consecutive input images. The current input image stored here is output to the ball selecting means 12.
The image storage unit 112 (first image storage means) stores the input image (odd field and even field) at the first position (past) among the three consecutive input images.

The difference image generation unit 113 (first difference image generation unit) delays the current input image, and subtracts the luminance of the newly input image (future) from the delayed luminance of the current input image. An image 1 (first difference image) is generated, and the generated difference image 1 is output to the candidate image generation means 115.
The difference image generation unit 114 (second difference image generation unit) delays the past input image, and subtracts the luminance of the current input image from the luminance of the delayed past input image, thereby calculating the difference image 2 (first image). 2 difference images) and the generated difference image 2 is output to the candidate image generation means 115.

  The candidate image generation unit 115 compares the difference image 1 and the difference image 2 with a predetermined luminance threshold for all the pixels in the search region. For example, when the pixel value of the image satisfies a predetermined condition, A binarized image is generated by discriminating the pixel value as “1 (white)”, otherwise “0 (black)”. Thereby, the candidate image generating unit 115 can extract, for example, an area where the pixel value is “1 (white)” as a video object candidate. The binarized image generated here is output to the ball selecting means 12 as an object candidate image obtained by extracting video object candidates.

  The ball selection unit 12 (object extraction unit) selects an extraction (tracking) target from the object candidate images generated by the object candidate image generation unit 11 based on the extraction conditions stored in the extraction condition storage unit 13. The video object (ball) is selected, and the position of the video object and the feature quantity characterizing the video object are extracted. Here, the ball selecting means 12 includes a labeling unit 121, a feature amount analyzing unit 122, a filter processing unit 123, and an object selecting unit 124.

  The labeling unit 121 assigns numbers (labels) to regions that are candidates for video objects in the object candidate images (binarized images) generated by the object candidate image generation unit 11. That is, the labeling unit 121 assigns one number to a connected area (connected area) having a pixel value of “1 (white)” that is an area of the video object. As a result, the video object candidates in the object candidate image are numbered.

For each video object candidate numbered by the labeling unit 121, the feature amount analysis unit 122 includes feature amounts (parameters) such as the position coordinates of the video object candidates and the “area, luminance, color, and circularity” of the video object. ) Is calculated.
Here, “position” indicates, for example, the barycentric position of the video object. “Area” indicates, for example, the number of pixels of the video object. “Luminance” indicates the average value of the luminance of each pixel in the video object. “Color” indicates an average value of RGB values of each pixel in the video object. The “circularity” indicates the circularity of the video object, and has a larger value as it becomes closer to a circle. For example, when the video object to be extracted has a circular shape such as a ball, the circularity of the extraction condition becomes a value closer to 1 as it is closer to a circle. The circularity e is expressed by the following equation (1), where S is the area of the video object and L is the perimeter.

e = 4πS / L ² (1)

The filter processing unit 123 uses the parameter value calculated by the feature amount analysis unit 122 to determine whether the video object matches the extraction condition information (parameter target value) stored in the extraction condition storage unit 13. Thus, the video objects to be extracted (tracked) are narrowed down. That is, for each video object candidate, the filter processing unit 123 extracts an extraction condition (for example, area, luminance, color, and circularity) stored in the extraction condition storage unit 13 and a predicted position by the position prediction unit 14 described later. The video object satisfying the extraction condition should be extracted by applying the feature value analyzed by the feature value analysis unit 122 to a filter (position filter, area filter, luminance filter, color filter, and circularity filter) based on Select as a video object.
The current field image (odd field and even field) stored in the image storage unit 111 is output to the ball selection unit 12 as described above, and the filter processing unit 123 converts the input image into one frame (2 fields). ) Filtering is performed at the “current” timing using the image delayed by the reference image as a reference image.

  The object selection unit 124 selects an object candidate closest to the predicted position coordinates of the ball as a ball among the objects that have passed through all the filters. The extracted position of the video object is written in the extraction condition storage unit 13 as current video object position information. Here, as the position of the video object, the barycentric coordinates of the video object, the vertex coordinates of the polygon approximation, the control point coordinates of the spline curve, and the like can be used. If the object selection unit 124 cannot select a video object that matches the extraction condition, the object selection unit 124 notifies the drawing / image composition unit 30 to that effect (extraction failure).

  The extraction condition storage means 13 stores conditions for selecting a video object to be extracted (tracked), and is a general storage medium such as a hard disk. The extraction condition storage unit 13 stores extraction condition information (parameter target value) indicating various extraction conditions and position information indicating the position of the video object.

The extraction condition information is information describing the extraction condition of the video object to be extracted, and describes, for example, at least one extraction condition of area, luminance, color, and circularity. This extraction condition information includes a filter (area filter, luminance filter, color filter, and circularity filter) for the ball selecting unit 12 to select a video object to be extracted from the object candidate image generated by the object candidate image generating unit 11. ).
The extraction condition information stores a predetermined initial value and a threshold value indicating an allowable range as conditions for the area filter, the luminance filter, the color filter, and the circularity filter. Thereby, each filter can exclude a video object having a value (feature) outside the threshold from candidates of video objects to be extracted.

  The position information stored in the extraction condition storage unit 13 is information indicating the position of the video object currently being tracked. This position information is, for example, the barycentric coordinates of the video object. The barycentric coordinates are calculated by the feature amount analysis unit 122 of the ball selection unit 12. This position information is also used as an extraction condition for determining the video object closest to the coordinates indicated by the position information as the video object to be extracted when there are a plurality of video objects that match the extraction condition information. Is done.

The position predicting means 14 estimates the predicted position of the ball based on the position (center of gravity coordinates, etc.) of the ball selected by the ball selecting means 12, and sets the predicted position information in the next input field image as a search region. This is output to the means 15.
The position predicting unit 14 includes a linear predicting unit 141, a curve predicting unit 142, and a switching unit 143.

  The linear predicting means 141 predicts that the ball trajectory is a straight line by using the motion vector. For example, the linear prediction unit 141 predicts the position of the ball in the next field image (next frame) by applying a Kalman filter to the barycentric coordinates, and estimates the search area. The Kalman filter changes from moment to moment by applying a recurrence formula that performs “filtering” to estimate the current state based on the observed quantity observed in time series and “prediction” to estimate the future state. The state to be performed is estimated.

The curve predicting unit 142 predicts that the ball trajectory is a quadratic curve obtained by the method of least squares. The curve predicting means 142 updates the quadratic curve (y = ax ² + bx + c) used for the curve prediction every time a ball is extracted, and monitors the coefficients a and b of the quadratic curve. It is determined whether or not the sign of b has changed or the values of the coefficients a and b have exceeded a predetermined value, and the determination result is output to the switching means 143.

The switching unit 143 switches between linear prediction by the linear prediction unit 141 and curve prediction by the curve prediction unit 142 based on the number of ball position coordinates extracted in the past or a quadratic curve equation. Here, for example, the switching unit 143 selects the curve prediction when the position coordinates extracted in the past of the video object are a predetermined number or more, and selects the linear prediction when the position coordinates are smaller than the predetermined number. The predetermined number is, for example, three, and five or more are preferable when obtaining an accurate curve. In addition, when curve prediction is executed, when the signs of the coefficients a and b of the quadratic curve equation (y = ax ² + bx + c) change or the values of the coefficients a and b exceed a predetermined value. The switching means 143 switches from curve prediction to linear prediction.

  The search area setting means 15 uses the predicted position of the ball in the next field (next frame) calculated by the position prediction means 14 to set a search area for the ball within a predetermined range in the field image. The search area setting means 15 outputs the set position and size of the search area to the object candidate image generation means 11 as search area information. The initial value of the search area is input by input means such as a mouse or a keyboard (not shown). The search area setting means 15 sequentially moves the initially set search area in accordance with the movement of the ball.

  The video object locus synthesizing apparatus 1 described above can be realized by causing a general computer to execute a program and operating an arithmetic device or a storage device in the computer. This program (video object trajectory synthesis program) can be distributed via a communication line, or can be distributed on a recording medium such as a CD-ROM.

Next, the operation of the video object locus synthesizing device 1 will be described with reference to FIGS. FIG. 2 is a flowchart showing the overall operation of the video object trajectory synthesis apparatus 1. FIG. 3 is a flowchart showing the operation of the ball extraction process by the video object extraction device 10. FIG. 4 is a flowchart showing the operation of object candidate image extraction processing in the object candidate image generating means 11. FIG. 5 is a flowchart showing the operation of the ball selection process in the ball selection means 12.
First, the operation of the video object trajectory synthesis apparatus 1 will be described with reference to FIG. 2 (refer to FIG. 1 as appropriate).

  The video object locus synthesizing device 1 performs a process of extracting a ball to be extracted (tracked) from the input video by the video object extracting device 10 (step S201). The detailed operation of the ball extraction process in step S201 will be described later with reference to FIG.

  The video object locus synthesizing apparatus 1 uses the drawing means 31 to draw an image indicating the feature amount of the video object (step S202). In step S202, a feature amount (for example, texture) is drawn at the position (for example, barycentric coordinates) of the ball extracted in step S201.

  Then, the video object trajectory synthesizing apparatus 1 synthesizes the trajectory image and the input image constituting the video delayed by the video delay means 20 by the image synthesizing unit 32 to generate a synthesized image (step S203). The image composition means 32 continuously outputs the composite image, thereby generating a composite video in which the moving trajectory of the video object is combined with the video.

Subsequent to step S203, the video object trajectory synthesis device 1 determines whether there is an image (next input image) to be processed next (step S204). If there is a next input image (step S204; Yes), the video object trajectory synthesis device 1 inputs next by the position predicting means 14 based on the position (center of gravity coordinates, etc.) of the ball extracted in step S201. The search area of the ball in the image to be estimated is estimated (step S205), and the process returns to step S201. As will be described later, the least square method and the Kalman filter can be used for the estimation of the search region. In addition, the video object trajectory synthesis apparatus 1 performs ball extraction while appropriately updating the area where the ball is searched, so that the amount of calculation for extracting / updating the ball can be suppressed. If there is no next input image in step S204 (step S204; No), the operation is terminated.
Through the above operations, the video object trajectory synthesis apparatus 1 sequentially extracts and tracks video objects (balls) to be tracked from field images input in time series as video, and synthesizes the moving trajectory of the video object into the video. The synthesized video can be output.

Next, the operation of the ball extraction process (step S201 in FIG. 2) in the video object extraction device 10 of the video object locus synthesis device 1 will be described with reference to FIG. 3 (refer to FIG. 1 as appropriate).
First, a search area is input by input means such as a mouse and a keyboard (not shown), and the video object extraction device 10 includes a range of a part of an image in an image constituting the video input by the search area setting means 15. A search area for searching for a video object is set in step S301. Note that the position of this search area is appropriately updated following the movement of the ball.

  Next, the video object extracting device 10 performs a process of extracting a ball candidate image (object candidate image) from the search area by the object candidate image generating unit 11 (step S302). The detailed operation of the ball candidate image extraction process in step S302 will be described later with reference to FIG.

When the candidate image for the ball is extracted by the object candidate image generating unit 11, the video object extracting device 10 narrows down the candidate image for the ball by the ball selecting unit 12, and extracts the position coordinates and the feature amount of the ball (step S303). ). The detailed operation of the ball selection process in step S303 will be described later with reference to FIG.
Subsequent to step S303, the video object extraction device 10 outputs the position coordinates and feature quantities of the ball selected and selected from the object candidate images in step S303 to the drawing / image composition unit 30 by the ball selection unit 12 (step S304). ).

Next, the operation of the object candidate image extraction process (step S302 in FIG. 3) in the object candidate image generation unit 11 of the video object extraction device 10 will be described with reference to FIG. 4 (refer to FIG. 1 as appropriate).
Three consecutive input images (past, present and future) are input to the object candidate image generating unit 11.

The object candidate image generation unit 11 uses the difference image generation unit 113 to determine the current input image (image stored in the image storage unit 111: for example an odd field) and the future input image (latest input image: for example an odd field). A difference image Image1 (difference image 1) is generated with the luminance difference between the pixel values as pixel values (step S401).
Further, the object candidate image generation means 11 is processed by the difference image generation means 114 with the current input image (image stored in the image storage unit 111: for example, an odd field) and the past input image (stored in the image storage unit 112). A difference image Image2 (difference image 2) having a pixel value as a luminance difference between the image and an odd number field is generated (step S402).

  The object candidate image generation unit 11 determines, using the candidate image generation unit 115, the areas where the difference images Image1 and Image2 satisfy the following conditions, and extracts them as object candidate images (step S403). In the present embodiment, the video object to be extracted (tracked) is a ball, and the luminance of the ball is generally higher than the luminance of the background image, so that the expressions (2) and (3) are simultaneously satisfied. When the conditional expression.

Image1 (x, y)> T (2)
Image2 (x, y) <− T (3)

Here, Image1 (x, y) in the above formulas (2) and (3) is the luminance value of the pixel located at the coordinates (x, y) of the difference image Image1, and Image2 (x, y) is the coordinates of the difference image Image2. The luminance value T of the pixel located at (x, y) is set as a luminance threshold.
The reason why an object (ball) that moves at high speed when the difference images Image1 and Image2 satisfy the conditional expressions (2) and (3) as described above will be described below with reference to FIG.

FIG. 6 is an explanatory diagram for explaining conditional expressions (2) and (3) of the difference image. FIG. 6A shows three consecutive input images (each of which is, for example, an odd field image). FIG. 6B shows two difference images. As shown in FIG. 6A, three consecutive field images 601, 602, and 603 representing the past, present, and future include one ball (M1, M2) at the past, present, and future positions. , M3). The balls M1, M2, and M3 are moving at a high speed that does not overlap the field images, and the luminance of the balls is higher than the luminance of the background image.
Further, as shown in FIG. 6B, the difference image Image1 is generated using a difference obtained by subtracting the luminance of the field image 603 from the luminance of the field image 602 as a pixel value. Further, the difference image Image2 is generated by using a difference obtained by subtracting the luminance of the field image 602 from the luminance of the field image 601 as a pixel value.

The region R2 in the difference image Image1 is configured with a luminance difference value obtained by subtracting the luminance of the region where no object exists from the luminance of the ball M2 as a pixel value, and thus the pixel value has a plus sign (for example, white).
The region R3 in the difference image Image1 is configured with a luminance difference value obtained by subtracting the luminance of the ball M3 from the luminance of the region where no object exists, so that the pixel value has a minus sign (for example, black).
The other regions including the region R1 in the difference image Image1 are configured by using the luminance difference value obtained by subtracting the luminances of the regions where no object exists as the pixel value, and thus the pixel value is 0.

Since the region R1 in the difference image Image2 is configured with a luminance difference value obtained by subtracting the luminance of the region where no object exists from the luminance of the ball M1 as a pixel value, the pixel value has a plus sign (for example, white).
The region R2 in the difference image Image2 is configured with a luminance difference value obtained by subtracting the luminance of the ball M2 from the luminance of the region where no object exists, so that the pixel value has a minus sign (for example, black).
The other regions including the region R3 in the difference image Image2 are configured by using the luminance difference value obtained by subtracting the luminances of the regions where no object exists as the pixel value, and thus the pixel value is 0.

When the difference image Image1 and the difference image Image2 are compared, the sign of the pixel value in the difference image Image1 is positive (formula (2)), and the sign of the pixel value in the difference image Image2 is negative (formula (3)) ) Area represents the area R2, that is, the ball M2 (current position). Therefore, the video object extraction device 10 can extract a video object (ball) moving at high speed as an object candidate image from the conditional expressions (2) and (3).
Note that when the luminance of the target video object is lower than the luminance of the background image, it is a condition that Expression (4) and Expression (5) are satisfied at the same time.

Image1 (x, y) <− T (4)
Image2 (x, y)> T (5)

  In this case, a region in which the sign of the pixel value is negative in the difference image Image1 (formula (4)) and the sign of the pixel value is positive in the difference image Image2 (formula (5)) is extracted as a video object. .

The description of the operation of the video object trajectory synthesis apparatus 1 in FIG. 1 will be continued.
The operation of the ball selection process (step S303 in FIG. 3) in the ball selection means 12 of the video object extraction device 10 will be described with reference to FIG. 5 (refer to FIG. 1 as appropriate).
The ball selection unit 12 uses the labeling unit 121 to assign a number (label) to a region that is a candidate for a video object in the object candidate image (step S501). The subsequent operations are processed in units of video objects based on the numbers assigned to the video object candidates.

  The ball selection means 12 is extracted by the feature value analysis unit 122 for each video object numbered in step S501 based on the extraction conditions stored in the extraction condition storage means 13 from the object candidate images. A video object to be tracked is selected, the position and feature amount of the selected video object are analyzed (calculated), and the position and feature amount of the video object are extracted (step S502). As the position of the video object, for example, the barycentric coordinates of the video object are used. As the feature amount of the video object, the area, luminance, color, circularity, etc. of the video object are used.

  The ball selection unit 12 selects a video object from the object candidate images based on the number (label) by the filter processing unit 123, and performs a filter process (step S503). That is, the filter processing unit 123 determines whether or not the “position” of the selected video object matches the range predicted by the position prediction unit 14. Here, when the matching condition based on the “position” is met, the filter processing unit 123 determines whether the “area” of the selected video object matches the extraction condition stored in the extraction condition storage unit 13. To do. Here, when the matching condition based on “area” is met, the filter processing unit 123 determines whether or not the “color” of the video object matches the extraction condition. If the matching condition based on “color” is met, the filter processing unit 123 determines whether the “luminance” of the video object matches the extraction condition. When the matching condition based on “luminance” is met, the filter processing unit 123 determines whether the “circularity” of the video object matches the extraction condition. Further, when the matching condition based on the “roundness” is met, that is, when all the extraction conditions are met, the filter processing unit 123 selects the previously selected video object as the video object to be tracked.

Subsequent to step S503, the ball selecting unit 12 performs a process of selecting the video object closest to the predicted position coordinates as the tracking target ball by the object selecting unit 124 (step S504). As a result, a ball can be selected even when there are a plurality of video objects that have passed through all the filters.
With the above operation, the video object extraction device 10 can select a video object (ball) suitable for the extraction condition from the object candidate images by the ball selection unit 12.

Next, the operation of the position prediction process (step S205) by the position prediction unit 14 of the video object extraction device 10 will be described with reference to FIG. FIG. 7 is an explanatory diagram for explaining the position prediction process.
It is assumed that balls M1 to M4 at different times are obtained by past object extraction processing, and that the position coordinates of the ball M5 (double circle in the figure) in the current field image are extracted. The position prediction means 14 uses the linear prediction means 141 to determine the motion vector indicated by the arrow V1 based on the extracted position coordinates (ball M5) in the current field and the extracted position coordinates (ball trajectory M4) in the previous field (previous frame). Ask for.

  Then, the linear prediction unit 141 adds a motion vector V2 similar to the motion vector V1 to the position of the ball M5 with respect to the extracted position coordinates (ball M5) in the current field based on the motion vector V1. As a predicted position of the ball, a linear predicted position coordinate Y1 (triangle mark in the figure) is temporarily determined (linear prediction). On the other hand, the position predicting unit 14 uses the curve predicting unit 142 to obtain a quadratic curve K1 indicated by a broken line that passes through the ball trajectory (balls M1 to M5) from the past to the present using the least square method. . Then, the position predicting unit 14 corrects the linear predicted position coordinate Y1 on the quadratic curve K1, and obtains a predicted curve position coordinate Y2 indicated by a square mark (curve prediction). The obtained curve predicted position coordinate Y2 is set as a search position in the next field image (next frame).

However, the position predicting unit 14 performs linear prediction by the linear predicting unit 141 and curve prediction by the curve predicting unit 142 based on the number of position coordinates of the ball extracted in the past by the switching unit 143 or a quadratic curve equation. Switch. That is, the position predicting unit 14 does not perform the curve prediction when there is not a sufficient number of past ball position coordinates (three but five or more are suitable) for executing the curve prediction. Perform linear prediction only. Further, the position predicting unit 14 changes the sign of the coefficients a and b of the quadratic curve (y = ax ² + bx + c) used for the curve prediction when the curve predicting unit 142 executes the curve prediction, When the values of a and b exceed a predetermined value, switching to linear prediction is performed.

  Next, an example in which the video object trajectory image composition device 1 composes the trajectory of the video object will be described with reference to FIG. FIG. 8 is an explanatory diagram for explaining an example in which the moving trajectory of the video object (ball) is synthesized on the video. In the following description, it is assumed that a baseball is tracked and the movement trajectory is synthesized.

The ball thrown by the pitcher PT toward the catcher CT is extracted with the position coordinates (10 points in the figure) at a predetermined timing, and is stored in the mitt of the catcher CT through the positions of the balls M1 to M10.
At this time, the video object locus synthesizing apparatus 1 predicts the position coordinates of the ball by the position prediction means 14 by linear prediction at first. When a trajectory with a sufficient number of points (for example, five points) for curve prediction is extracted, the position prediction unit 14 switches from linear prediction to curve prediction. Here, the left batter BT is interposed between the balls M6 and M7 shown in FIG. When the ball hits the uniform of the left batter BT, the difference between the luminance of the ball and the luminance of the uniform is so small that the ball is blinded and the extraction of the ball may fail. Even in such a case, the video object locus synthesizing apparatus 1 can predict the position of the ball M7 based on the movement locus of the ball going back several points from the ball M6 by executing the curve prediction. Accordingly, even when the extraction of the ball is blinded by the moving object (left batter BT), the predicted position of the ball after exceeding the left batter BT is the previous field compared with the case where only the linear prediction is performed. It is connected smoothly from the (front frame) and becomes accurate. Therefore, the video object trajectory synthesis apparatus 1 can narrow the search area, so that the amount of calculation required for ball extraction can be kept low.

  In the video object locus synthesizing apparatus 1, as described above, when the ball extraction fails, the ball selecting unit 12 notifies the drawing / image synthesizing unit 30 to that effect. In this case, the drawing / image combining means 30 calculates the ball position by interpolation from the position (for example, the center of gravity) of the balls M6 and M7 on the trajectory image by the interpolation means (not shown). N1 can be drawn.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment. For example, in the present embodiment, the video object has been described as a baseball ball, but may be a ball used in other sports such as a soccer ball, a tennis ball, and a golf ball. Furthermore, the video object is not limited to a ball as long as it moves at a high speed that does not cause overlapping of field images. In this embodiment, a field is used as the unit time of the input image, but other time units such as a frame may be used.

It is the block diagram which showed the structure of the video object locus | trajectory synthesis apparatus based on this invention. It is a flowchart which shows the whole operation | movement of the video object locus | trajectory synthesis apparatus which concerns on this invention. It is a flowchart which shows the operation | movement of the ball | bowl extraction process by the video object extraction device which concerns on this invention. It is a flowchart which shows the operation | movement of the extraction process of the object candidate image in an object candidate image generation means. It is a flowchart which shows the operation | movement of the ball selection process in a ball | bowl selection means. It is explanatory drawing for demonstrating the conditional expression of a difference image, (a) is three continuous field images, (b) is two difference images. It is explanatory drawing for demonstrating a position prediction process. It is explanatory drawing for demonstrating the example which synthesize | combined the movement locus | trajectory of the video object on the image | video.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Video object locus synthesis apparatus 10 Video object extraction apparatus 11 Object candidate image generation means 12 Ball selection means (object extraction means)
DESCRIPTION OF SYMBOLS 13 Extraction condition memory | storage means 14 Position prediction means 15 Search area setting means 20 Image | video delay means 30 Drawing and image composition means 31 Drawing means 32 Image composition means 111 Image storage part (2nd image storage means)
112 Image storage unit (first image storage means)
113 difference image generation means (first difference image generation means)
114 difference image generation means (second difference image generation means)
115 Candidate Image Generation Unit 121 Labeling Unit 122 Feature Quantity Analysis Unit 123 Filter Processing Unit 124 Object Selection Unit 141 Linear Prediction Unit 142 Curve Prediction Unit 143 Switching Unit

Claims (6)

A video object extraction device that extracts a position of a video object included in an input video,
An object candidate image generating means for generating an object candidate image extracted as a candidate for the video object from a continuous first image, second image and third image input in time series;
Object extraction means for extracting the position of the video object to be extracted from the generated object candidate images based on a predetermined extraction condition;
The object candidate image generation means includes
First image storage means for storing the first image;
Second image storage means for storing the second image;
First difference image generation means for generating a first difference image having a pixel value as a luminance difference between the third image and the second image;
Second difference image generation means for generating a second difference image having a pixel value as a luminance difference between the second image and the first image;
A sign of a pixel value of the first difference image and a sign of a pixel value of the second difference image with reference to a pixel value arranged at a common position of the first difference image and the second difference image; Discriminating different regions, and candidate image generating means for generating the object candidate image,
A video object extraction device comprising:   Based on the position of the video object extracted by the object extraction means, the position of the video object in the next input image is predicted, and the predicted position information of the video object is output to the object candidate image generation means. 2. The video object extracting device according to claim 1, further comprising an object position predicting unit. The object position predicting means includes
Linear prediction means for predicting that the trajectory of the video object is a straight line;
Curve prediction means for predicting that the trajectory of the video object is a curve;
Switching means for switching between linear prediction by the linear prediction means and curve prediction by the curve prediction means based on the number of position coordinates of the video object extracted in the past or the equation of the curve;
The video object extracting device according to claim 2, wherein: The video object extraction device according to any one of claims 1 to 3,
A trajectory in which a moving trajectory of the video object is extracted by drawing an image showing the video object in an area corresponding to an image input in time series based on the position of the video object extracted by the object extracting means. A drawing means for generating an image;
Image synthesizing means for synthesizing the trajectory image and the image input in time series;
A video object locus synthesizing apparatus comprising: A video object trajectory synthesis method that extracts and tracks a video object included in an input video and synthesizes a moving trajectory of the video object on the video,
Of the first image, the second image, and the third image that are input in time series, a first difference image having a pixel value as a luminance difference between the third image and the second image A difference image generation step for generating and generating a second difference image having a difference in luminance between the second image and the first image as a pixel value;
With reference to the pixel value arranged at the common position of the first difference image and the second difference image generated in the difference image generation step, the sign of the pixel value of the first difference image and the second difference image An object candidate image generation step of generating an object candidate image obtained by extracting an area having a different pixel value code as the video object candidate;
An object extraction step of extracting the position of the video object from the object candidate image generated in the object candidate image generation step based on at least one extraction condition of position, area, luminance, color, and circularity;
Based on the position of the video object extracted in this object extraction step, a trajectory in which the moving trajectory of the video object is extracted by drawing an image showing the video object in an area corresponding to the image input in time series A drawing step for generating an image;
An image synthesis step of synthesizing the trajectory image generated in this drawing step and the image input in time series;
A video object trajectory synthesis method characterized by comprising: In order to extract and track a video object included in the input video, and to synthesize a moving locus of the video object on the video,
Of the first image, the second image, and the third image that are input in time series, a first difference image having a pixel value as a luminance difference between the third image and the second image Differential image generating means for generating a second differential image with a pixel value as a luminance difference between the second image and the first image;
The sign of the pixel value of the first difference image and the pixel of the second difference image with reference to the pixel value arranged at the common position of the first difference image and the second difference image generated by the difference image generating means Object candidate image generation means for generating an object candidate image obtained by extracting a region having a different sign from the value as a candidate for the video object;
Object extraction means for extracting the position of the video object from the object candidate image generated by the object candidate image generation means based on at least one extraction condition of position, area, luminance, color and circularity;
Based on the position of the video object extracted by the object extracting means, a trajectory in which the moving trajectory of the video object is extracted by drawing an image showing the video object in an area corresponding to the image input in time series Drawing means for generating images,
Image synthesis means for synthesizing the trajectory image generated by the drawing means and the image input in time series,
A video object trajectory synthesis program characterized by functioning as

Images