KR19990086247A - Sports visual effect processor - Google Patents

Abstract

The present invention relates to a sports image effect processing apparatus, and in particular, the image processing unit 3 comprises a swing motion extracting unit 31 and a ball tracking unit 32, the swing motion extracting unit 31 is a foreground recording unit ( 311), the background recording unit 312, the object area extraction unit 313, the player area extraction unit 314 and the swing output control unit 315 to extract each motion area of the athlete extracted on one screen in a superimposed or unfolding method Synthetic expression allows the viewer to easily analyze and understand the overall swinging motion of the golf player, the ball tracking unit 32 is a tracking object selection unit 321, tracking window setting unit 322, initial background image generation unit 323 ), An input image noise canceller 324, an object extractor 325, a tracking center setting unit 326, a composite background image generator 327, and a tracking output control unit 328 are obstructed when tracking a ball. To overcome the background noise of athlete's feet or sports equipment objects and increase the reliability of tracking. It is.

Description

Sports visual effect processor

The present invention relates to a sports image processing apparatus, and more particularly, in sports broadcasting, a player can accurately capture a player through a video extraction and synthesis method in a golf game, a baseball game, a bowling game or a tennis game where swing time is short and the posture is important. By extracting and expressing the swing motions sequentially, the invention is invented so that ordinary viewers can easily analyze and understand the swing motions of the athlete.

Conventionally, a magnetic club is attached to a golf club, baseball bat, bowling glove, tennis racket, etc. to analyze a golf player's tee shot swing, a swinging posture of a baseball player, a bowling player's pitching form, and a tennis player's form. Using a sensor and a computer to detect the location of golf clubs or baseball bats, bowling gloves and tennis rackets, analyze the swing motion or install infrared equipment in a pre-set location to play golf clubs, baseball bats, bowling gloves and tennis rackets There is a device that detects when the back light passes infrared rays and measures the swing speed.

However, attaching specific devices to golf clubs, baseball bats, bowling gloves, tennis rackets, etc. as described above interferes with the progress of each sporting event, and the use of infrared devices is difficult to install and flexible to changes in player position. There is a disadvantage that can not cope.

Therefore, the method of using such a separate measuring device is limited and it is difficult to apply to the actual broadcast relay broadcast.

In addition, chroma key synthesis and matte synthesis are used to extract specific objects from an image input from a camera.

By the way, the former chroma key synthesis method is a method of extracting a specific object using color information from an image photographed against a special screen (mostly blue) called a chroma key back. It is difficult to apply to the actual game due to the restriction that it should be used only within the chroma key bag.

In addition, the latter matting method is a method of directly specifying a contour of an object to be extracted by a human hand using a mouse or a tablet, and thus can be applied to all images without requiring a special background. However, there is a problem in that it takes a lot of work time because a manual operation is required for every frame for moving images.

Accordingly, the present invention has been made in order to solve the above-mentioned conventional problems, the image input from the camera without additional measuring equipment for analyzing the swing motion of a golf player, baseball player, bowling player or tennis player, etc. The main purpose of the present invention is to provide a sports image processing apparatus that can extract and analyze only the athlete's area to analyze and correct the swing motion of the athlete.

Another object of the present invention is to superimpose or unfold the extracted athlete area to express the swing motions of consecutively inputted athletes on one screen so that the motion can be easily understood and can be effectively applied to sports broadcasting. An effect processing apparatus is provided.

Still another object of the present invention is to provide a sports visual effects processing apparatus that can increase the reliability of tracking a ball for various races by applying a tracking directionality table when tracking a ball used in various races.

The object of the present invention is to capture a situation of various races and to output video data; An A / D converter converting analog data input from the camera into digital for image processing; An image processor for extracting a desired signal through a predetermined algorithm with respect to the digital image signal continuously input through the A / D converter; An image and parameter storage unit configured to store processing result information and parameters in real time in the image processor; It is composed of D / A converting unit that separates the final extracted object image and the background image from the digital image data output from the image control unit and converts the digital image obtained by combining the final extracted object image and the background image into an output unit. A sports image processing apparatus comprising: a swing motion extracting unit extracting only a player from a digital image signal continuously input by using a player extraction algorithm to extract swing motions of players in various races; It consists of a ball tracking unit that extracts the ball from the continuous digital input signal using the tracking algorithm to be used for the ball visualization, and the image and parameter storage unit and the reference point of the player image and the player area extracted from the image processing unit To save parameters such as spatial location information in real time,

The swing motion extracting unit records and smoothes only a background image; A foreground recording unit for recording and smoothing a player's continuous swing motion; An object region extraction unit using a difference image between the reference image selected by the background recording unit and the input image in the foreground recording unit; A player area extraction unit for removing external background noise from the rough object area output from the object area extraction unit and separating a background and a player; A player area extraction unit for removing background noises around the object area after basic image processing and obtaining only pure player images; A track output control unit configured to control the output of the extracted player image, the ball tracker comprising: a tracker selector configured to initially select a tracker from an image input from a camera; A tracking window setting unit for setting a window around the selected track object; An initial background image generator for generating a pure background image excluding a traced object within a tracking window setting range; An input image noise removing unit for removing the overall noise from the image input from the camera unit; An object extracting unit extracting a trace object using the input image output from the initial background image and the input image noise removing unit; A tracking center setting unit for generating a new tracking window by obtaining the center of gravity of the extracted object, and a synthesis background for creating a background image for extracting a tracking object of the next input image within the new tracking window range set by the tracking center setting unit. An image generator; It can be achieved by configuring a tracking output control unit for screen control for the composite image using the result region and the parameter of the object extraction unit.

Therefore, during the sports broadcast broadcast where the swing time is short and the posture is important, the player can accurately extract the player's swing movements by sequentially extracting and synthesizing the video through the video extraction and synthesis method, and the general viewer can easily analyze the swing movement of the athlete. And understandable.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in detail.

1 is an overall block diagram of an apparatus of the present invention

FIG. 2 is a block diagram illustrating a swing motion extractor of FIG. 1.

3 is a block diagram illustrating the hole extracting unit in FIG. 1.

4 (a)-(d) are exemplary views of a screen displayed on the monitor as a result of swing motion extraction.

5 is an exemplary diagram for generating an initial background synthesis image in a tracking window according to the present invention.

6 is a configuration of a tracking directionality table for setting an expected tracking direction of a golf ball according to the present invention.

7 (a)-(d) are exemplary views of a screen displayed on a monitor as a result of tracking a ball (eg golf ball) by the device of the present invention.

Explanation of symbols for main parts in the drawings

1: camera 2: A / D conversion unit

3: image processing unit 4: image and parameter storage unit

5: D / A conversion unit 6: Output unit

31: swing motion extraction unit 32: ball tracking unit

311: foreground recording unit 312: background recording unit

313: object region extraction unit 314: bow region extraction unit

315: swing output control unit 3131: difference video unit

3132: region labeling unit 3133: manual editing unit

321: tracking object selection unit 322: tracking window setting unit

323: Initial background image generating unit 324: Input image noise removing unit

325: object extraction unit 326: tracking center setting unit

327: synthetic background image generation unit 328: tracking output control unit

3251: difference image portion 3252: region labeling portion

3253: tracking direction table application unit 3254: morphological processing unit

1 is a block diagram illustrating the entire block diagram of the apparatus of the present invention, FIG. 2 is a block diagram illustrating the swing motion extraction unit in FIG. 1, FIG. 3 is a block diagram illustrating the hole extraction unit in FIG. 1, and FIG. 4. (A)-(d) shows an exemplary view of a screen appearing on the monitor as a result of swing motion extraction, and FIG. 5 shows an exemplary view for generating an initial background composite image in a tracking window according to the present invention. 6 shows a configuration of a tracking directional table for setting an expected tracking direction of a golf ball according to the present invention, and FIGS. 7 (a) to 7 (d) show ball (eg, golf ball) tracking by the present invention. As a result, an example of the screen appearing on the monitor is shown.

According to this, the camera 1 which captures the situation of various competitions, and outputs image data; An A / D converter 2 for converting analog data input from the camera 1 into digital for image processing; An image processor (3) for extracting a desired signal through a predetermined algorithm with respect to the digital image signal continuously inputted through the A / D converter (2); An image and parameter storage unit 4 for storing the processing result information and the parameters of the image processing unit 3 in real time; After separating the background image and the object image from the digital image data output from the image processing unit (3) converts the final extracted object image and background image by converting the digital image into an analog signal to output to the output unit (6) In the sports image processing apparatus composed of the D / A converter (5),

The image processing unit (3) includes a swing motion extracting unit (31) for extracting only the athletes from the digital image signals continuously input by using the player extraction algorithm to extract the swing motions of the athletes in various races; It consists of a ball tracking unit 32 for extracting the ball from the continuously input digital image signal using the tracking algorithm to be used for the visualization of the ball, the image and parameter storage unit 4 is the image processing unit (3) ), And the parameters such as the reference point and the spatial position information of the athlete's area extracted from the athlete's area are stored in real time.

At this time, the swing motion extractor 31 includes a foreground recording unit 311 for smoothing the input image to capture the swing motion of the athlete from the output of the camera 1 and to remove the image noise; A background recording unit 312 for shooting and smoothing a pure background image without a player while the camera is fixed; An object region extraction unit 313 for extracting an object block through a difference image between a reference image selected by the background recording unit 312 and an input image by the foreground recording unit 311; A bow region extractor 314 for removing external background noise from the rough object region output from the object region extractor 313 and separating a background and a bow; And a swing output control unit 315 for controlling input and output of the player area extracted from the player area extraction unit 314 and various types of information.

On the other hand, the object region extraction unit 313 is a difference image unit 3131 for extracting and binarizing the difference components of the foreground image and the background image output from the foreground recording unit 311 and the background recording unit 312, respectively; An area labeling unit 3132 for dividing the binarized image output from the difference image unit 3131 into area units and classifying a region close to the center of the screen among large areas as an object area; It is characterized in that it comprises a manual editing unit (3133) for manually editing the part of the exercise machine object that is easy to damage in the object region generated as a result of the area labeling unit (3132).

In addition, the ball tracking unit 32 includes a tracker selection unit 321 for allowing a user to initially select an object to be tracked in an image input from the camera 1 with a mouse; A tracking window setting unit 322 for setting a tracking window centering on the tracking object selected by the tracking object selecting unit 321; An initial background image generator 323 for generating a pure background image excluding a traced object within a tracking window range set by the tracking window setting unit 322; An input image noise removing unit 324 for removing an overall noise from an image signal input from the camera 1; An object extracting unit 325 for extracting a trace using an initial background image output from the initial background image generating unit 323 and an input image output from the input image noise removing unit 324; A tracking center setting unit 326 for obtaining a center of gravity of the object extracted by the object extracting unit 325 and generating a new tracking window; And a synthetic background image generator 327 for generating a background image for extracting a trace of a next input image within a new tracking window range set by the tracking center setting unit 326: the object extractor 325. The tracking output control unit 328 performs screen control for the synthesized image using the result region and the parameter of FIG.

In this case, the object extractor 325 is a difference image between the background image output from the initial background image generator 323 or the composite background image generator 327 and the input image output from the input image noise canceller 324, respectively. A difference image unit 3251 to obtain and generate a binarized image thereof; An area labeling unit 3252 for dividing the binarized image output from the difference image unit 3251 in area units; A tracking directionality table applying unit 3253 using directional information of the tracker to remove tracking noises such as a foot of an athlete or a predetermined area of an object of a sports equipment among the areas set as the object zone resulting from the area labeling unit 3252. ; Characterized in that it consists of a morphological processing unit 3254 for removing the background noise for the trace output from the tracking direction table application unit 3253.

Referring to the operation and effect of the device of the present invention configured as described above in detail.

First, the image signal input from the camera 1 is input to the image and parameter storage unit 4 via the A / D change unit 2 and the image processor 3.

At this time, the image input from the camera 1 is directly stored in the memory of the workstation in order to reduce the processing time and the image is processed in the image extraction and tracking algorithm, and the image is stored in the image and parameter storage unit 4 in real time.

The video and parameter storage unit 4 is a real-time external disk having a transmission speed of 40 MB / sec and a speed of 30 frames / sec for an uncompressed video data, which is suitable for sports relay broadcasting. The image stored in the image is called out and sent through the D / A change unit 5.

The swing motion extractor 31 of the image processor 3 records a swing motion of a player (for example, a golf player or a baseball player) from the camera 1 and uses a 3x3 smoothing filter to remove the image noise. 311 and the image output respectively output from the background recording unit 312 which records only the background image and removes the image noise, obtains the difference image of these images from the difference image unit 3131 and binarizes them.

Since the background components of the pixel components of the input image and the background image will have similar brightness, the difference component has a low value and the portion where the player appears in the input image has a relatively high value.

In the case of obtaining the difference image using only the luminance component, the distortion may occur in the object region at a low contrast ratio between the input image and the background image. By using the brightness components of the G and B pixels, distortion in the object region that may occur when generating the difference image is reduced.

That is, a composite background image is created for each of the R, G, and B images, and the largest difference component among the difference values between the R, G, and B images of the current image in each pixel is determined as the object pixel.

In the present invention, the binarization threshold value is 10, and the pixel value of the difference image is determined as an object, and the pixel value of the difference value or less is determined as a background component, and the difference image is binarized into an object and a background.

As a result of the binarization, there are many error regions that are incorrectly recognized as object regions even though the image is an actual background component, and according to the present invention, the region labeling unit 3132 numbers each region of the binarized image and is smaller than the threshold pixel size. The areas were raised.

In this case, the threshold for object zone classification is arbitrarily set according to the size of the object to be extracted.

And for the background noise area larger than this critical pixel, using the athlete's spatial position information, the large area of the object deviating from the center of the screen was regarded as the background component and the player's object area to be removed and extracted.

In a result region of the area labeling unit 3132, a predetermined portion (eg, a head of a golf club) of the exercise device object may be separated and removed from the object area due to the high speed swinging operation.

At this time, the manual editing unit 3133 connects the exercise device area and the object area with a mouse while the user views the entire areas, thereby preventing the loss of a predetermined area of the exercise device area.

In addition, the bow region extraction unit 314 fills holes that may be inside the object region by performing morphological processing, which is one of image processing techniques, to remove the background noise stuck around the object region of the area labeling unit 3132. After removing external background noise, various kinds of information about the athlete's area are generated.

Equation (1) below shows a morphological process for filling a hole inside the object area and removing background noise stuck around the object area.

Erosion (3, A, B)

Dilation (3, A, B) (1)

Dilation (5, A, B)

Erosion (5, A, B)

Where A is the object region and B is the mask block for morphological processing, and the number 3 or 5 represents the size of the mask.

Various parameters of the area extracted by the player area extraction unit 314 are the size of the area, the coordinates of the minimum boundary rectangle surrounding the player area, the temporal position of the current input image in the input video sequence, the start and end points of the swing, and the arrangement of the players. Reference point.

Here, the array reference point is set to arrange the extracted player area at a specific spatial position when synthesizing the images. For example, the lower right foot of the player area with less movement in golf swing motion is set as the array reference point.

The swing output control unit 315 controls the video output by using the player area and various parameters generated as a result of the player area extraction unit 314.

On the other hand, Figure 4 (a)-(d) is an exemplary view of a screen that appears on the monitor the results of golf swing motion extraction by the apparatus of the present invention, (a) and (b) shows the extracted player area on one screen Overlapping expressions, in which (a) is an overlapping expression for one player and (b) is an overlapping expression for comparing two players.

4C and 4D show the unfolded expression of the extracted player area.

This is an expression of unfolding the athlete at regular intervals starting from the first reference point of the arrangement of the athlete's area, where (c) is the unfolding expression of one athlete and (d) is the unfolding expression for comparing two athletes.

As shown in (a)-(d) of FIG. 4, the method of extracting a player and visualizing it on one screen in expressing a swing motion of a golf player can see the swing motion of the player at a glance, so that the swing motion of the player is shown to the viewer in detail. It can be used for posture correction and for golf practitioners.

Meanwhile, the ball tracking unit 32 (for example, the golf ball tracking unit) first selects an object to be tracked by the mouse on the object to be tracked with the mouse on the input image by drawing a rectangle with the mouse.

In this case, the tracking window setting unit 322 sets a tracking window having a size three times larger than a rectangle surrounding the selected tracking object and removes image noise using a 3x3 smoothing filter within this range.

The initial background image synthesis unit 323 fills the selected trace object selection rectangle with an external background to create a pure background image.

5 shows the shape of the tracking window, the outermost rectangle is the outer range of the tracking window, block 5 is the rectangle surrounding the selected track, and blocks of the same size except block 5 are background blocks.

In this case, the object block 5 is replaced with a block having an intermediate value of pixel average values of the background blocks to form an initial composite back image.

The input image noise removing unit 324 removes the image noise by using the 3x3 smoothing filter at the same position as the tracking window selected by the tracking window setting unit 322 for the image input from the camera 1.

In addition, the difference image unit 3251 and the region labeling unit 3252 of the object extractor 325 may include the difference image unit 3131 and the region labeling unit used by the object region extractor 313 of the swing motion extractor 31. 3132).

The tracking direction table application unit 3253 predicts the direction of the initial golf ball by using a method of finding a real ball, for example, a golf ball area, from among several areas resulting from the area labeling unit, and converts the object area within the direction of the golf ball into the golf ball. Choose.

6 shows the tracking direction table of the golf ball.

The entire input image is divided into sections (a) to (b), and the expected tracking direction of the golf ball is determined by which section the starting point of the initial golf ball is located.

Equation (2) below is the expected running direction of the golf ball corresponding to each section.

Section (a): 270。 <θ <30。

Section (b): 180。 <θ <0。

Section (c): 150。 <θ <270。

Section (d): 270。 <θ <90。

Section (e): 0。 <θ <360。 (2)

Section (f): 90。 <θ <270。

Section (g): 330。 <θ <90。

Section (h): 0。 <θ <180。

Section (i): 90。 <θ <210。

By using the ball's expected direction table, tracking errors, for example, caused by golf club and player instep, can be eliminated.

The ball block extracted by the tracking directionality table removes background noise from a morphological processing unit 3254 such as the player area extraction unit 314 of the swing motion extraction unit 31.

At this time, the final hole area obtains the center of gravity of the area in the tracking center setting unit 326, and selects a section three times the size of the initially selected object area as a new tracking window.

In addition, the composite background image generator 327 creates a composite background image by replacing the empty area extracted from the new tracking window section with the background area at the position of the previous frame.

In this way, the input image can obtain the difference between the background image and the previous background image.

In addition, the tracking output control unit 328 performs screen control for the composite image by using square hole parameters such as the area size, spatial coordinates, the center of gravity, the starting point and the end point of the tracking, and the starting point and the end point of the input image.

7 (a)-(d) are exemplary diagrams of a screen in which a tracking state of a golf ball is displayed on a monitor by the apparatus of the present invention, and (a) shows a result of the tracking as a simple trajectory. b) encloses the trace with an ellipse mark at the same time as the trajectory, (c) shows the trajectory of the tracking process following the line, and (d) shows the trajectory of the golf ball flying at high speed.

As described above, according to the apparatus of the present invention, the trajectory of the ball (that is, the golf ball, the baseball ball, the bowling ball, the tennis ball, etc.) can be seen at a glance, increasing the visible effect to the viewer, and applying the directional table to track the ball. Can increase the reliability.

As described above, according to the present invention, the motions that are easily missed in the high-speed golf swing motions continuously input through the swing motion analysis for the broadcast broadcast and the method of tracking and expressing the balls are extracted by superimposing each motion of the golf player. Or, by reorganizing on one screen with spread expression, you can observe the start and end of the movement at a glance, and you can compare the swing motion of two players at the same time, so that ordinary viewers can easily understand and analyze, and the direction of the ball It is possible to express visually and to easily overcome the external background noise such as golf clubs and human feet, for example, by using the spatial directional table of the ball.

Claims (6)

A camera 1 for capturing the situation of various races and outputting image data; An A / D converter 2 for converting analog data input from the camera 1 into digital for image processing; An image processor (3) for extracting a desired signal through a predetermined algorithm with respect to the digital image signal continuously inputted through the A / D converter (2); An image and parameter storage unit 4 for storing processing result information and parameters of the image processor 3 in real time; After separating the background image and the object image from the digital image data output from the image processing unit (3) converts the final extracted object image and background image by converting the digital image into an analog signal to output to the output unit (6) In the sports image processing apparatus composed of the D / A converter (5), The image processing unit 3 includes a swing motion extracting unit 31 which extracts only a player from a digital video signal continuously input by using a player extraction algorithm to extract a swing motion of a player playing each ball; It consists of a ball tracking unit 32 for extracting the ball from the continuously input digital image signal using the tracking algorithm to be used for the visualization of the ball, the image and parameter storage unit 4 is the image processing unit (3) Sports image effect processing apparatus characterized in that the parameters such as the player image and the reference point and the spatial position information of the athlete region extracted from the) in real time. The method of claim 1, wherein the swing motion extracting unit 31 includes a foreground recording unit 311 for smoothing the input image to capture the swing motion of the athlete from the output of the camera 1 and to remove the image noise; A background recording unit 312 for shooting and smoothing a pure background image without a player while the camera is fixed; An object region extraction unit 313 for extracting an object block through a difference image between a reference image selected by the background recording unit 312 and an input image by the foreground recording unit 311; A player region extraction unit 314 for removing external background noise from the rough object region output from the object region extraction unit 313 and separating a background and a player: a player region extracted from the player region extraction unit 314; Sports image effect processing apparatus comprising a swing output control unit 315 for controlling the input and output of various information. The method according to claim 1, wherein the ball tracking unit 32 and the track object selection unit 321 for allowing the user to initially select the object to be tracked in the image input from the camera (1); A tracking window setting unit 322 for setting a tracking window centering on the tracking object selected by the tracking object selecting unit 321; An initial background image generator 323 for generating a pure background image excluding a traced object within a tracking window range set by the tracking window setting unit 322; An input image noise removing unit 324 for removing the overall noise from the image signal input from the camera 1; An object extracting unit 325 for extracting a trace using an initial background image output from the initial background image generating unit 323 and an input image output from the input image noise removing unit 324; A tracking center setting unit 326 for obtaining a center of gravity of the object extracted by the object extracting unit 325 and generating a new tracking window; A composite background image generator 327 for generating a background image for extracting a trace of a next input image within a new tracking window range set by the tracking center setting unit 326; Sports image effect processing apparatus, characterized in that consisting of a tracking output control unit (328) for expressing the result image synthesized using the result region and the parameter of the object extraction unit (325). 3. The difference image unit 3131 of claim 2, wherein the object region extraction unit 313 extracts a difference component of the foreground image and the background image output from the foreground recording unit 311 and the background recording unit 312, respectively, and binarizes them. Wow ; An area labeling unit 3132 for dividing the binarized image output from the difference image unit 3131 into area units and classifying a region close to the center of the screen among large areas as an object area; And a manual editing unit (3133) for manually editing an object part of a sports equipment that is easily damaged in the object region generated as a result of the area labeling unit (3132). The method according to claim 2, The swing output control unit 315 is a variety of information extracted by the player area extraction unit 314, namely the arrangement reference point of the player area used as the screen arrangement criteria when the screen display, the player area number, the size of the player area and In the resynthesis of the athlete's movements using the total number of athlete's zones, the position of the start point and the end point in the screen display, the swing movements of the athletes are superimposed on the arrangement reference points of the extracted athlete zones, or the first A sports visual effects apparatus, comprising: setting a reference point of an array of player blocks at a random position and setting the starting point and synthesizing the next player blocks by unfolding them at a predetermined interval. The input image output from the background image and the input image noise canceller 324 are respectively output from the initial background image generator 323 or the composite background image generator 327. A difference image unit 3251 to obtain a difference image of the resultant and generate the binarized image; An area labeling unit 3252 for dividing the binarized image output from the difference image unit 3251 in area units; In addition to the ball area to be tracked among the various areas extracted as a result of the area labeling unit 3252, a tracking direction table application unit 3253 using directional information of the tracker to remove tracking noise such as a foot of a player or an object of a sports equipment. )Wow ; Sports morphology processing unit, characterized in that configured as a morphological processing unit (3254) for removing the background noise for the trace output from the tracking direction table application unit (3253).