JP2005209148A - Offside line detector, program therefor, and player location detector - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an offside line detector which detects the position of an offside line from an image obtained by photographing a soccer game. <P>SOLUTION: An offside line detector 1 is provided with; a silhouette image generation means for dividing an input image into player areas and a background area; a three-dimensional player coordinate estimation means 12 for estimating three-dimensional estimate player coordinates of players; a two-dimensional player area setting means 13 for setting two-dimensional player areas of players in the input image; a color statistic measuring means 14 for measuring statistics of colors in two-dimensional player areas; a color statistic storage means 15 for storing statistics of colors based on jersey colors of teams to which players belong; a classifying means 16 for classifying players into teams on the basis of statistics of colors; and an offside line position output means for specifying and outputting the position of an offside line on the basis of a team to which each player belongs, and three-dimensional player estimate coordinates. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a CG (Computer Graphics) synthesis technique for sports video, and more specifically, an offside line detection device and program for detecting the position of an offside line for use in CG synthesis from a soccer game video, and The present invention relates to a player position detection device.

2. Description of the Related Art Conventionally, as a CG synthesis technique for sports video, an apparatus that virtually visualizes a 10-yard line that is a unit of attack in American football on a video has been put into practical use. In addition, in order to make it easy to see the flight distance of tee shots in golf, there is a device that virtually superimposes a graphic such as a straight line on a video at a specific distance on a video. Further, in swimming and speed skating, there is a technique for visualizing a past swimmer or runner's actual video superimposed on the current competition video.
Furthermore, a technique for detecting the position of a sports athlete (player) by attaching a tag for electromagnetic reading to a sports athlete, and visualizing an offside line of a sports competition is disclosed based on the position (Patent Document) 1, see Patent Document 2).
Japanese Patent Laid-Open No. 7-56990 (paragraphs 0016 to 0028, FIGS. 1 to 3) JP 2001-104533 A (paragraphs 0017 to 0027, FIG. 1)

  However, in the conventional sports video CG composition technology, for example, when a 10-yard line in American football is superimposed on a video, the 10-yard line is a static physical quantity during the attack period of one team, and the period Among them, the CG indicating the 10 yard line need only be fixedly superimposed on the video. In addition, when the golf flight distance is superimposed on the video as a scale, the scale is a completely static physical quantity. For example, a fixed scale such as a 10-yard interval or a 50-yard interval is superimposed on the video as a CG. do it.

On the other hand, when an actual video of a past swimmer or runner is superimposed on a current competition video, as in the case of swimming or speed skating, the CG data to be handled (actual video of past swimmer or runner) is dynamic. Although it is a thing, it is a past physical quantity accumulated in advance.
That is, the conventional sports video CG composition technique has a problem that physical quantities that change in real time cannot be superimposed on the video, such as an offside line in a soccer game.

  In addition, by attaching a tag for electromagnetic reading to sports athletes, the technology for visualizing off-side lines of soccer competitions, etc. must attach tags to sports athletes (players), and handling is troublesome. There is a problem that it gives a sense of foreign material to the material feeling of the uniform and puts a burden on the players. Furthermore, there is a problem that the tag may be damaged due to an impact caused by the fall of the player, causing malfunction.

  The present invention has been made in view of the above problems, and a player position detection device capable of detecting the position of a dynamically changing player for each team from a video obtained by photographing a sports competition; To provide an offside line detection device capable of dynamically detecting the position of an offside line in a soccer game by detecting the position of a player for each team without attaching a tag for electromagnetic reading, and a program thereof. Objective.

  The present invention was devised in order to achieve the above-mentioned object. First, the player position detection device according to claim 1 detects the position of a player for each team from an input video obtained by photographing a sports competition with a camera. An apparatus for detecting a position of a player, comprising: a silhouette video generation means; a three-dimensional player coordinate estimation means; a two-dimensional player area setting means; a color statistic measurement means; a color statistic storage means; and a classification means. The configuration.

  According to this configuration, the player position detection device generates a silhouette video divided into the player area and the background area in time series from the input video by the silhouette video generation means. For example, the input video is binarized by color information (color vector or the like) of the input video. Or it is good also as binarizing an input image | video by detecting the area | region which has a motion for every flame | frame as a player area. Thus, a candidate area for the player area is extracted from the input video.

  Then, the player position detection device uses the three-dimensional player coordinate estimation means to determine the player area based on the position of the player area in the silhouette video (for example, the center of gravity position), the position where the camera is installed, and the focal length at the time of shooting. The position of the player is estimated by tracking the position of the player on the three-dimensional coordinates. That is, the three-dimensional player coordinate estimation means back-projects a two-dimensional image captured by the camera onto the real space (three-dimensional coordinates) based on the position and focal length of the camera, and the player region on the three-dimensional coordinates. By tracking the position of the player, the position of the player is estimated as three-dimensional estimated player coordinates. As a result, the position of the player on the input video in the real space is specified. Moreover, even if the player areas overlap with each other and have the same silhouette, the overlap (hiding) can be detected. At this stage, the teams of players are not distinguished.

Further, the player position detecting device projects and converts the three-dimensional estimated player coordinates for each player region estimated by the three-dimensional player coordinate estimating unit based on the position of the camera and the focal length by the two-dimensional player region setting unit. Then, it is converted into two-dimensional coordinates, and a two-dimensional player area where the player exists, for example, a rectangular area, is set in the input video according to a predetermined player size. Thereby, the player area specified by the individual three-dimensional estimated player coordinates is associated with the two-dimensional player area.
Then, the player position detection device detects the color of the input video in the common area between the two-dimensional player area set by the two-dimensional player area setting means and the player area in the silhouette video by the color statistic measuring means. Measure statistics. In this way, by measuring the color statistics in the common area between the two-dimensional player area and the player area in the silhouette video, even if the players overlap as silhouettes, the colors can be displayed as different players. It becomes possible to measure the statistic.

  In addition, the player position detection device includes a color statistic storage unit that stores at least a color statistic based on the clothes color of the team to which the player belongs. For this reason, the player position detection device uses the classification means to determine the player based on the color statistics stored in the color statistics storage means and the color statistics measured by the color statistics measurement means. It becomes possible to classify for each team to which the player belongs.

  Furthermore, the player position detection device according to claim 2 is the player position detection device according to claim 1, wherein the three-dimensional player coordinate estimation means includes an area determination means, and is determined to be effective by the area determination means. For the player area, the three-dimensional estimated player coordinates are estimated.

  According to such a configuration, the player position detection device determines whether or not the player area in the silhouette video is effective as the player area based on the size of the player determined in advance by the area determination means in the three-dimensional player coordinate estimation means. Determine. That is, even when the area determination unit extracts a region similar to the player's color as a player region during binarization by the silhouette video generation unit, the player has a predetermined size (area) of the region. If it is not included in the size range, it is determined that the area is not a player area.

  The player position detecting device according to claim 3 is the player position detecting device according to claim 1 or 2, wherein the three-dimensional player coordinate estimating means includes a three-dimensional coordinate converting means and a second 2 A three-dimensional player area setting means, an area threshold value generating means, and a variable area determination means, wherein the three-dimensional estimated player coordinates are estimated for a player area determined to be valid by the variable area determination means. .

  According to such a configuration, the player position detection device converts the two-dimensional coordinates for each player area in the silhouette video into three-dimensional coordinates based on the position and focal length of the camera by the three-dimensional coordinate conversion means. The player position detecting device is based on the three-dimensional coordinates converted by the second two-dimensional player region setting means by the three-dimensional coordinate conversion means, the position and focal length of the camera, and a predetermined player size. Then, a two-dimensional player area where a player exists in the input video is set. Further, the area threshold generation unit generates a threshold for determining whether or not the player area is effective as the player area based on the size of the two-dimensional player area. That is, by setting a predetermined width for the size of the two-dimensional player area, the upper limit value and the lower limit value of the threshold are set.

As a result, the threshold for determining whether or not the player area is valid as the player's area changes according to the size of the player even if the size of the player changes on the video. Thresholds can be set.
Then, the player position detection device determines whether or not the player area in the silhouette video is valid as the player area based on the threshold value generated by the area threshold value generation means by the variable area determination means. This makes it possible to accurately extract an effective area in the player area.

  Furthermore, the player position detection device according to claim 4 is the player position detection device according to any one of claims 1 to 3, wherein the three-dimensional player coordinate estimation means is a detection / tracking means; A prediction / estimation unit and a delay unit are provided.

  According to this configuration, the player position detection device uses the three-dimensional predicted player coordinates that are three-dimensional coordinates of the player area predicted by the detection / tracking means in the past by one unit time and the three-dimensional coordinates of the current player area. A player area is detected and tracked for each player based on a distance from a certain three-dimensional player coordinate. For example, the detection / tracking means can track the player by determining that the one having the shortest distance between the three-dimensional predicted player coordinate and the three-dimensional player coordinate is the same player region. Further, when the distance is long, it is possible to detect a new player area by determining that a player has newly appeared in the video.

  Then, the player position detecting device detects the player tracking means by the prediction / estimating means and smoothes the tracked three-dimensional player coordinates for each player in the time direction to obtain the three-dimensional estimated player coordinates. The three-dimensional coordinates of the player area after unit time are predicted as the three-dimensional predicted player coordinates. Here, the smoothing in the time direction is to convert the three-dimensional player coordinates input for each time into a smooth movement locus in consideration of noise and the like. For example, the prediction / estimation means functions as a Kalman filter to convert it into a smooth movement locus that takes noise into consideration.

  Then, the player position detection device delays the three-dimensional predicted player coordinates predicted by the prediction / estimation unit by one unit time by the delay unit. Here, the delayed three-dimensional predicted player coordinates are used in the prediction / estimation means. Accordingly, the player position detection device can track the player and can accurately identify the player position even when the player areas overlap.

  The player position detection device according to claim 5 is the player position detection device according to any one of claims 1 to 3, wherein the color statistic storage means stores a statistic of a referee's clothing color. In this case, the classification means sets the referee in addition to the player as a classification target.

  According to this configuration, the player position detection device can recognize the referee separately from the player by storing the statistic of the referee's clothing color in the statistic storage means. This makes it possible to detect not only the player but also the position of the referee in the video.

  Moreover, the offside line detection apparatus of Claim 6 is an offside line detection apparatus which detects an offside line from the input image | video which image | photographed the soccer game with the camera, Comprising: The any one of Claims 1 thru | or 4 The player position detecting device described in 1 and offside line position output means are provided.

  According to this structure, an offside line detection apparatus estimates a player's position as a three-dimensional estimated player coordinate from an input image | video with a player position detection apparatus. Then, a two-dimensional player area corresponding to the three-dimensional estimated player coordinates is specified, and a color statistic is measured for the two-dimensional player area (two-dimensional player area), thereby matching the players and teams. To detect. Thereby, it is possible to detect the position (three-dimensional estimated player coordinates) of the player shown on the input video in association with the team to which the player belongs. And the offside line detection apparatus can pinpoint the position of an offside line by the offside line position output means based on the three-dimensional estimated player coordinate of each player in an input image, and the team to which a player belongs. For example, the offside line position output means sets the line parallel to the goal line as the position of the offside line at the position of the second player from the back of the defensive side including the goalkeeper.

  Furthermore, the offside line detection device according to claim 7 is the offside line detection device according to claim 6, wherein the offside line position output unit includes a coordinate storage unit and an offside line coordinate estimation unit. .

  According to this configuration, the offside line detection device stores the three-dimensional estimated player coordinates in association with the type information for identifying the team in the coordinate storage unit of the offside line position output unit. As a result, all the positions of the players in the frame at a certain time are stored. Then, the offside line detection device uses the offside line coordinate estimation unit to determine the size of a specific coordinate component (for example, x coordinate) of the players divided into teams in the three-dimensional estimated player coordinates stored in the coordinate storage unit. In this order, the value of the predetermined coordinate component is estimated as the position of the offside line. For example, when the coordinate system is closer to the goal line on the home team side as the x coordinate is larger, the position of the player having the second largest x coordinate can be estimated as the position of the offside line.

  An offside line detection device according to claim 8 is an offside line detection device that detects an offside line from an input video obtained by shooting a soccer game with a camera, and detects the positions of a player and a referee. The player position detection device described above and offside line position output means are provided, and the offside line position output means includes coordinate storage means and offside line coordinate estimation means.

  According to this configuration, the offside line detection device estimates the positions of the player and the referee as three-dimensional estimated player coordinates from the input video by the player position detection device. The offside line detection device stores the three-dimensional estimated player coordinates in association with the type information for identifying the player and the referee in the coordinate storage unit of the offside line position output unit. Thereby, all the positions of the player and the referee in the frame at a certain time are stored. The offside line detection device is based on the type information and the order of the size of the specific coordinate component in the specific team in the three-dimensional estimated player coordinates stored in the coordinate storage unit by the offside line coordinate estimation unit. Then, the value of the coordinate component that is a predetermined number by the player or the referee is estimated as the position of the offside line. For example, if the coordinate system is closer to the goal line on the home team side as the x coordinate is larger, the position of the referee is further increased even if the position of the player with the second largest x coordinate is estimated as the position of the offside line. When it is close to the goal line, it is assumed that the referee is tracking a player who is not shown in the video, and the position of the referee is estimated as the position of the offside line.

  Furthermore, the offside line detection device according to claim 9 is the offside line detection device according to claim 7 or 8, wherein the offside line detection device is predetermined among the three-dimensional estimated player coordinates detected by the player position detection device. And a filter means for storing in the coordinate storage means only the coordinates existing within the specified range.

  According to such a configuration, the offside line detection device stores the three-dimensional estimated player coordinates detected by the player position detection device within the predetermined range by the filter means, and stores the coordinates in the range. Is not stored in the coordinate storage means. For example, when the goal keeper is more than a predetermined distance from the goal, the goal keeper is not likely to be erroneously detected, and the coordinates are not stored in the coordinate storage means. As a result, only the position of the player or the like for detecting the offside line is stored in the coordinate storage means.

  The offside line detection device according to claim 10 is an offside line detection device for detecting an offside line from an input video obtained by shooting a soccer game with a camera, and includes a silhouette video generation unit and a three-dimensional object coordinate estimation unit. And a two-dimensional player area setting means, a color statistic measurement means, a color statistic storage means, a classification means, and an offside line position output means.

According to such a configuration, the offside line detection device generates a silhouette video divided into an object region and a background region indicating a player or a ball in time series from the input video by the silhouette video generation unit.
Then, the offside line detection device uses the three-dimensional object coordinate estimation unit to determine the object region based on the position of the object region in the silhouette video (for example, the center of gravity position), the position where the camera is installed, and the focal length at the time of shooting. The position of the player or the ball is estimated by tracking the position of the player on the three-dimensional coordinates.
Further, the offside line detection device projects and converts the player's three-dimensional estimated player coordinates estimated by the three-dimensional object coordinate estimating means based on the position and focal length of the camera by the two-dimensional player area setting means. Converted into two-dimensional coordinates, a two-dimensional player area, for example, a rectangular area, in which the player exists is set in the input video according to a predetermined player size. Then, the offside line detection device detects the color of the input video in a region common to the object region in the silhouette video and the two-dimensional player region set by the two-dimensional player region setting unit by the color statistic measurement unit. Measure statistics.

The offside line detection device further includes a color statistic storage unit that stores a color statistic based on at least the clothing color of the team to which the player belongs. For this reason, the player position detection device uses the classification means based on the color statistics stored in the color statistics storage means and the color statistics measured by the color statistics measurement means, It becomes possible to classify for each team to which the player belongs.
Then, the offside line detection device specifies the position of the offside line by the offside line position output means based on the three-dimensional estimated player coordinates of each player in the input video, the team to which the player belongs, and the ball position coordinates. can do. For example, when the ball is not reflected in the input image, the position of the offside line is estimated from the position of the player, and when the ball is reflected, the position of the ball is estimated as the position of the offside line.

  Furthermore, the offside line detection device according to claim 11 is the offside line detection device according to claim 10, wherein the color statistic storage means stores the statistic of the referee's clothing color. Classification means classifies the referee as a position detection target in addition to the player, and the offside line position output means includes coordinate storage means and offside line coordinate estimation means.

According to this configuration, the offside line detection device can recognize the referee separately from the player by storing the statistic of the referee's clothing color. This makes it possible to detect not only the player but also the position of the referee in the input video.
The offside line detection device stores the three-dimensional estimated player coordinates and the three-dimensional estimated ball coordinates in the coordinate storage unit of the offside line position output unit in association with the type information identifying the player, the referee, and the ball. Then, the offside line detection device uses the offside line coordinate estimation means to determine a predetermined coordinate component for the player, the referee, or the ball based on the type information and the order of the sizes of specific coordinate components (for example, the x coordinate). Is estimated as the position of the offside line.

  Furthermore, the offside line detection program according to claim 12, the computer is used to detect an offside line from an input video obtained by photographing a soccer game with a camera, a silhouette video generation means, a three-dimensional player coordinate estimation means, a two-dimensional It functions as a player area setting means, a color statistic measurement means, a classification means, and an offside line position output means.

By causing the computer to function in this manner, the offside line detection program generates a silhouette video divided into a player area and a background area in time series from the input video by the silhouette video generation means.
Then, the offside line detection program uses the three-dimensional player coordinate estimation means to determine the position of the player area based on the position of the player area (for example, the center of gravity position) in the silhouette video and the position and focal length of the camera. By tracking above, the player's position is estimated.
Furthermore, the offside line detection program uses the two-dimensional player area setting means to calculate the three-dimensional estimated player coordinates for each player area calculated by the three-dimensional player coordinate estimation means, the camera position and focal length, Based on the size, a two-dimensional player area where a player exists in the input video is set.

Then, the offside line detection program detects the color of the input video in the common area between the two-dimensional player area set by the two-dimensional player area setting means and the player area in the silhouette video by the color statistic measuring means. Measure statistics.
The offside line detection program includes a color statistic stored in a color statistic storage unit that stores at least a color statistic based on a clothing color of a team to which the player belongs, and a color statistic measurement step. Players are classified by team based on the color statistics measured in.
Further, the offside line detection program uses the offside line position output means based on the team for each player classified by the classification means and the three-dimensional estimated player coordinates estimated by the three-dimensional player coordinate estimation means. Identify the location.

  According to the first aspect of the present invention, it is possible to detect and output the position of the player in the three-dimensional space for each team from the input video obtained by photographing the sports competition with the camera, and further, the position of the player for each frame. Therefore, the position of the player can be accurately detected even if the players overlap each other.

  According to the second aspect of the present invention, when the size (area) of the region extracted as the silhouette video is not included in the predetermined player size range, it is determined that the region is not a player region. Therefore, a region due to noise or erroneous detection can be eliminated, and a player can be detected with high accuracy.

  According to the third aspect of the present invention, even if the size (area) of the region extracted as a silhouette video changes greatly depending on the movement of the player, it is extracted as a silhouette video depending on the size of the player. Since the threshold value for determining whether or not the area is effective as the player area can be changed, the validity of the player area can be accurately determined.

According to the invention described in claim 4, since the player area can be detected and tracked on the three-dimensional coordinates, it is possible to track individual players even when the players overlap in the silhouette video. It becomes possible. Thus, since the players on the video can be distinguished, the position of the offside line can be accurately detected.
According to the fifth aspect of the present invention, the position of the referee can be detected and tracked on the three-dimensional coordinates, like the player.

  According to the invention described in claim 6 or claim 12, it changes from moment to moment according to the position in the three-dimensional space of each team player in the soccer competition field from the input video obtained by photographing the soccer competition with the camera. The position (coordinates) of the offside line can be estimated. Accordingly, the offside line can be visualized and presented by synthesizing the position (coordinates) of the offside line with the input video by computer graphics.

According to the seventh aspect of the present invention, for each time-series video (such as a frame), the position of the player shown in the video is classified and stored for each team. It is possible to estimate how many players from a certain team are located from the goal, and according to the order, the position of the offside line can be estimated.
According to the invention described in claim 8, like the player, the position of the referee can be detected and tracked on the three-dimensional coordinates, and each time-series video (frame etc.) is reflected in the video. Since the players and the referees are classified and stored, the position of the offside line can be estimated more accurately than when the position of the offside line is estimated only by the player.

According to the ninth aspect of the present invention, only the position of a player or the like necessary for detecting the offside line is stored in the coordinate storage means, so that the accuracy of estimating the position of the offside line is improved. Can do.
According to the invention described in claim 10, an offside line that changes from moment to moment depending on the position of the player and the ball in the three-dimensional space of each team in the soccer competition field from the input video obtained by photographing the soccer competition with the camera. Can be estimated.
According to the eleventh aspect, since the position of the offside line is estimated based on the position of the player, the referee, and the ball, the position of the offside line can be estimated with high accuracy.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Outline of offside line detector]
First, an outline of an offside line detection device according to the present invention will be described with reference to FIG. FIG. 1 is an explanatory diagram for explaining an outline of an offside line detection device that detects an offside line defined in a soccer competition rule.
As shown in FIG. 1, the offside line detection device 1 is based on an image output from a camera C that captures a soccer field F, a position where the camera C is installed, and a focal distance at the time of shooting. The position of the line is detected. Here, the offside line is a line parallel to the goal line GL passing between the half way line HL and the goal line GL and passing the second player from the rear side of the defensive side including the goal keeper.
Here, the offside line detection device 1 has the touch line TL direction in the soccer field F as the x (x _W ) axis, the half way line HL direction as the y (y _W ) axis, and the zenith direction as the z (z _W ) axis. The position of the offside line when the world coordinate system (x _W , y _W , z _W ) is set, that is, the x coordinate is output.

The camera C is assumed to be arranged at the camera position T (t _x , t _y , t _z ) in the world coordinate system (x _W , y _W , z _W ), and the coordinate system of the camera C is It is assumed that the camera coordinate system (x _C , y _C , z _C ) is different from the world coordinate system (x _W , y _W , z _W ). Here, the camera coordinate system is a coordinate system whose origin is a position that becomes a viewpoint when viewing an input video with respect to world coordinates. Further, the camera C does not need to be a fixed camera, and the soccer stadium F may be photographed with a moving camera.

<First embodiment>
[Configuration of offside line detector]
Next, the configuration of the offside line detection device according to the first embodiment of the present invention will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the offside line detection apparatus according to the first embodiment of the present invention.
As shown in FIG. 2, the offside line detection device 1 detects the position (coordinates) of the offside line by detecting the position of the player for each team from the video (input video) taken of the soccer game. is there. Here, the offside line detection apparatus 1 includes a player position detection unit 10 and an offside line position output unit 20.

  The player position detecting means (player position detecting device) 10 detects the position of the player for each team in the input video from the input sports video (here, the input video obtained by shooting a soccer game). Here, the player position detection means 10 includes a silhouette video generation means 11, a three-dimensional player coordinate estimation means 12, a two-dimensional player area setting means 13, a color statistic measurement means 14, and a color statistic storage means 15. The classification means 16 is provided.

  The silhouette video generation means 11 generates a silhouette video by dividing the frame into a plurality of player areas and a background area for each frame input in time series as an input video. Here, the silhouette video generation means 11 binarizes the frame with the player area having the pixel value “1” and the background area having the pixel value “0”. For example, the silhouette video shown in FIG. 4 is generated by binarizing the input video (frame at a certain time) shown in FIG. In FIG. 4, the player area is indicated by “black” and the background area is indicated by “white”. FIG. 4 shows an example in which a region H that is not related to a player is erroneously detected in the player region. The silhouette image generated here is temporarily stored in a storage means (not shown) and is referred to by the three-dimensional player coordinate estimation means 12 and the color statistic measurement means 14.

  The silhouette video generation means 11 can be realized by a general hard chroma key, for example. Here, when the input video is a color video, the pixel value I (x, y) at the (x, y) coordinate position of the input video takes a vector value (color vector) in the color space. Therefore, here, it is assumed that the input color vector is converted into a binary of {0, 1} as a hard chroma key. That is, the hard chroma key can be a function c that converts the input video I (x, y) into a silhouette video (binary image) S (x, y) as shown in the following equation (1).

  For example, statistics such as the average value and covariance of color vectors in the background lawn and soil area are obtained in advance, and “0” is obtained when the color vector of the pixel of the input video is close to the statistics. In other cases, the function c that outputs “1” is defined as a hard chroma key. Here, the Mahalanobis distance can be used as the distance between the color vector and the statistic. When the Mahalanobis distance is less than or equal to a predetermined threshold, the function c in the above equation (1) outputs “0”, and when it exceeds the threshold, the function c outputs “1”.

  The silhouette video generation means 11 can also be realized by a background difference method. For example, a background image B (x, y) in which a player is not shown or a player is removed is prepared in advance, and an input video I (x, y) as shown in the following equation (2): The silhouette video is set to “0” if the difference in color vector size from the background image B (x, y) prepared in advance is equal to or smaller than a predetermined threshold θ, and “1” otherwise. (Binary image) S (x, y) may be used.

  Note that the silhouette video generation means 11 may detect an area with movement for each frame and set the area with movement as a player area.

  The three-dimensional player coordinate estimation means 12 is based on the position of the player area in the silhouette video generated by the silhouette video generation means 11, the position where the camera that shot the input video is installed, and the focal length at the time of shooting, respectively. The three-dimensional estimated player coordinates obtained by estimating the three-dimensional coordinates of the player area are calculated. The three-dimensional player coordinate estimation means 12 determines the three-dimensional estimated player coordinates based on the position of the center of gravity which is a specific point in each player area. In addition to the barycentric coordinates, a specific point in the player area, for example, the highest point can be used, but it is preferable to use barycentric coordinates that characterize the position of the entire player area. The three-dimensional estimated player coordinates estimated by the three-dimensional player coordinate estimating means 12 are given an identification number for each player area and are output to the two-dimensional player area setting means 13. Here, the three-dimensional player coordinate estimation means 12 includes a labeling means 12a, an area determination means 12b, a back projection conversion means 12c, a detection / tracking means 12d, a prediction / estimation means 12e, and a delay means 12f. It is configured with.

  The labeling unit 12a performs numbering (labeling) on the player area in the silhouette video generated by the silhouette video generation unit 11. For example, by labeling the silhouette video shown in FIG. 4, as shown in FIG. 5, a label number (here, L1 to L9) is given to each player area that is a single connected area. . This labeling can be performed by a general binary image labeling algorithm. Here, it is assumed that the labeling means 12a also calculates the area and barycentric coordinates of the single connected area (player area) for each labeled label number. Hereinafter, this label number, area, and barycentric coordinates are referred to as region information. The region information that is information generated by the labeling unit 12a is output to the area determining unit 12b.

The area determination unit 12b functions as an area filter that determines whether the area in the region information generated by the labeling unit 12a is within a specific range and outputs only the region information whose area is in the specific range. Is. For example, the lower limit value A _min and the upper limit value A _max of the area corresponding to the size of the player are determined in advance as threshold values. Then, the area determining unit 12b, the label number m (1 ≦ m ≦ M) (M is the number of simply connected regions) area A _m of the area information corresponding to is included in the closed interval [A _min, A _max] The region information is output only when A _m ∈ [A _min , A _max ] is satisfied. Here, the area information that satisfies the condition is output to the backprojection conversion means 12c.
Accordingly, the area information of the area H that is not related to the player shown in FIG. 4 is not output.

The backprojection conversion unit 12c converts the center-of-gravity coordinates, which are two-dimensional coordinates for each player area included in the area information output from the area determination unit 12b, into three-dimensional coordinates (three-dimensional player coordinates in the world coordinate system illustrated in FIG. ). Here, the center-of-gravity coordinates are (g _x , g _y ), the camera position in the world coordinate system is (t _x , t _y , t _z ), the rotation matrix from the camera coordinate system to the world coordinate system is R, and the camera focus When the distance is f and the virtual height of the player in the world coordinate system is h, the three-dimensional player coordinates (w _x , w _y , w _z ) in the world coordinate system corresponding to the barycentric coordinates are expressed by the following equation (3): Can be sought.

The calculated three-dimensional player coordinates (w _x , w _y , w _z ) according to the equation (3) are output to the detection / tracking means 12d in association with the player area, that is, the label number.

The detection / tracking means 12d has three-dimensional player coordinates for each player area converted into the coordinate position of the world coordinate system by the back projection conversion means 12c, and 3 for each player area delayed and output by the delay means 12f described later. Based on the distance (for example, Euclidean distance) from the dimension predicted player coordinates, the player area is tracked in association with each player.
In this detection / tracking means 12d, the distance between the 3D player coordinates and the 3D predicted player coordinates for each player area is calculated in all combinations, and the 3D player coordinates of the closest distance to the 3D predicted player coordinates are calculated. The player area is determined to be the same player area as the three-dimensional predicted player coordinates, and the same identification number is assigned.
That is, the detection / tracking unit 12d firstly selects the three-dimensional player coordinates w _m (w _x , w _y , w _z) closest to the three-dimensional predicted player coordinates p _i of the player area of the identification number i delayed by the delay unit 12f. ) Label number k (i) is obtained by the following equation (4). M _f is a set of label numbers m of area information that satisfies the condition (filtered) by the area determination unit 12b.

Here, the distance d (i ₎ between the three-dimensional player coordinates w _{k (i) of} the label number k (i) and the three-dimensional predicted player coordinates p _i of the player area of the identification number i shown in the following equation (5). ) Is equal to or less than a predetermined threshold value, the detection / tracking means 12d determines that the tracking is successful, and predicts / estimates the identification number i and the three-dimensional player coordinates w _{k (i)} as a pair. To 12e. Thus, the same player area (player area) on the video is given the same identification number and is tracked.

  On the other hand, the detection / tracking unit 12d sets J as a set of identification numbers i in which the distance d (i) is equal to or less than a predetermined threshold, and relates to all the identification numbers i output from the delay unit 12f. After obtaining the nearest label number k (i) by the above, the set N of label numbers that have not become the nearest neighbor of any identification number i (εJ) is obtained by the following equation (6).

Here, when the set N in the expression (6) is not an empty set, the detection / tracking unit 12d assigns a new identification number not included in the set J to all the label numbers n (∈N) constituting the set N. allocation, and outputs to the prediction-estimating unit 12e and a 3-dimensional players coordinate w _n of the new identification number and label number n as a pair. As a result, a new player appearing on the video is detected, and a new identification number is assigned to the player's area (player area).

  The prediction / estimation means 12e smoothes the three-dimensional player coordinates for each player (player area) newly detected or tracked by the detection / tracking means 12d in the time direction to obtain the three-dimensional estimation player coordinates. In addition, the three-dimensional coordinates of the player area after one unit time are predicted as the three-dimensional predicted player coordinates. Here, the smoothing in the time direction is to convert the three-dimensional player coordinates input for each time into a smooth movement locus in consideration of noise or the like by filtering. The three-dimensional predicted player coordinates predicted by the prediction / estimation unit 12e are output to the delay unit 12f together with an identification number that identifies the player region. The three-dimensional estimated player coordinates estimated here are output to the two-dimensional player region setting unit 13 and the offside line position output unit 20 together with the identification number for specifying the player region.

  The prediction / estimation unit 12e can be configured as a Kalman filter, for example. Here, a prediction and estimation technique using a Kalman filter performed by the prediction / estimation unit 12e will be described with reference to mathematical expressions. Here, the description will be given focusing on the player with one identification number i in the three-dimensional player coordinates input to the prediction / estimation means 12e. In the following description, as shown in FIG. 1, the three-dimensional coordinate system uses the touch line TL direction in the soccer field F as the x axis, the half way line HL direction as the y axis, and the zenith direction as the z axis. Assume a right-handed system. Further, in the Kalman filter, both the x and y components of the three-dimensional coordinates and the unit time difference of x and y are set in the internal state, and modeling is performed using a constant velocity motion model.

Here, P is the state covariance that is the covariance between the internal state and the system noise, R is the observation noise covariance matrix that is the covariance of the observation noise mixed during observation, and the process noise covariance matrix is the covariance of the process noise. Is Q, H is the observation matrix representing the relationship between the internal state and the observation state, and F is the state transition matrix, the Kalman gain K _{j at} time j is calculated by the following equation (7). T indicates transposition.

Then, the prediction / estimation means 12e sequentially executes the internal state update and propagation calculations according to the following equations (8) and (9). W _j indicates the three-dimensional player coordinates at time j.

Then, the prediction / estimation unit 12e calculates the three-dimensional estimated player coordinate s _j and the three-dimensional predicted player coordinate p _j at time j by the following equation (10). I represents a unit matrix, and O represents a zero matrix.

Note that x _{j | j} and P _{j | j} shown in the equations (7) to (10) indicate the internal state of the prediction / estimation means 12e (Kalman filter) and are initialized by the following procedure. And
First, it is assumed that the three-dimensional player coordinate w _j0 is input to the prediction / estimation means 12e for the first time at time j ₀ . At this time, the prediction / estimation unit 12e sets x _{j0 | j0-1 according} to the following equation (11).

On the other hand, for the state covariance P, for example, a diagonal matrix as shown in the following equation (12) can be used. Note that σ _x , σ _y , σ _u , and σ _v are positive integers.

Then, after time (j ₀ +1), the prediction / estimation means 12e sequentially executes the calculations of the expressions (7) to (10) to obtain the three-dimensional estimated player coordinates s _j and the three-dimensional predicted player coordinates. p _j is calculated.
In the equations (7) to (9), the observation matrix H, the state transition matrix F, the process noise covariance matrix Q and the observation noise covariance matrix R are expressed by the following equations (13) to (16): To do.

Note that λ in the equation (16) may be a variable having a positive value, and the value may be changed according to the position of the player. For example, paying attention to the label number k (i) in the above equation (4), when j satisfying k (i) = k (j) exists in addition to j = i, the prediction / estimation means 12e It is determined that concealment has occurred, and the value of λ is increased. When j is only j = i, the prediction / estimation unit 12e determines that no concealment between the players has occurred, and decreases the value of λ. Accordingly, the prediction / estimation unit 12e can track the players with high accuracy in consideration of concealment between the players.
Thus, it becomes possible to track a player smoothly by comprising the prediction and estimation means 12e by a Kalman filter.
The description of the configuration of the offside line detection device 1 in FIG. 2 will be continued.

  The delay means 12f delays the three-dimensional predicted player coordinates for each player area after one unit time predicted by the prediction / estimation means 12e by one unit time, that is, one frame. The three-dimensional predicted player coordinates delayed here are output to the detection / tracking means 12d together with an identification number for specifying the player region.

The two-dimensional player area setting means (projection conversion means) 13 is the three-dimensional estimated player coordinates calculated by the three-dimensional player coordinate estimation means 12, the position and focal length of the camera that captured the input video, and the predetermined player's Based on the size, a two-dimensional player area, which is a two-dimensional area where a player exists in the input video, is set.
The two-dimensional player area setting means 13 converts a player area based on the three-dimensional coordinates (three-dimensional player coordinates) in the world coordinate system into a two-dimensional coordinate area (two-dimensional player area) in the input video. Is. The two-dimensional player area setting means 13 approximates each player's area based on the three-dimensional player coordinates as a cylinder, projects the cylinder onto the two-dimensional coordinates, and includes a rectangular area including the projected area ( Hereinafter, the bounding box) is set as the player's two-dimensional player area. The two-dimensional player area (bounding box) set here is output to the color statistic measuring means 14 together with the identification number.

Hereinafter, a bounding box setting method in the two-dimensional player region setting unit 13 will be described with reference to mathematical expressions. Here, the three-dimensional player coordinates in the world coordinate system are w = (w _x , w _y , w _z ), the camera position in the world coordinate system is t = (t _x , t _y , t _z ), and from the world coordinate system The rotation matrix to the camera coordinate system is R ^T , the camera focal length is f, the player's virtual height in the world coordinate system is defined as h, the virtual height of the player who approximates the cylinder is h, and the radius of the cylinder is r.
At this time, the perspective projection formula T (w) for converting the three-dimensional player coordinate w in the world coordinate system into the two-dimensional coordinate can be expressed by the following formula (17).

Therefore, the two-dimensional player region setting means 13 has coordinates w ₁ and w ₂ shifted by a radius r in the x-axis direction and coordinates w ₃ and w ₄ shifted by a radius r in the y-axis direction with respect to the three-dimensional player coordinate w. And the coordinates w ₅ and w ₆ shifted by ½ of the virtual height h in the z-axis direction are calculated by the following equations (18) to (23).

Further, the two-dimensional player region setting means 13 converts the coordinates w _{1 to} w ₆ calculated by the equations (18) to (23) to w in the equation (17) as shown in the equation (24). By substituting, the coordinates w _{1 to} w ₆ are projected and converted into two-dimensional coordinates.

  The two-dimensional player area setting means 13 has a range from the minimum value to the maximum value in the x-axis direction among the two-dimensional coordinates calculated by the expression (24), as shown in the expression (25), A region included in the range from the minimum value to the maximum value in the y-axis direction is defined as a bounding box D.

The description of the configuration of the offside line detection device 1 in FIG. 2 will be continued.
The color statistic measurement unit 14 includes a bounding box (two-dimensional player region) set by the two-dimensional player region setting unit 13 and a player region (for example, a pixel value “1”) of the silhouette image generated by the silhouette image generation unit 11. The color statistic of the input video is measured within a common area with the "" area). Here, the measured color statistic is output to the classification means 16 together with the identification number.
As the color statistics, for example, an average color vector obtained by averaging color vectors within a region can be used. That is, if the area common to the bounding box and the player area of the silhouette video (pixel value “1”) is E and the color vector in the (x, y) coordinates is I (x, y), the average color vector μ is , (26).

  This causes each bounding box to be characterized by a color statistic.

  The color statistic storage means 15 is a database that stores in advance color statistics based on at least the clothes color of the team to which the player belongs. Here, the color statistic storage means 15 stores the object type (type information) and the color statistic in pairs. Here, the object type is an identifier indicating to which team a person belongs, whether it is a goalkeeper, a referee, or the like. The object type and color statistics stored in the color statistics storage unit 15 are referred to by the classification unit 16.

Here, the database stored in the color statistic storage unit 15 will be described with reference to FIG. FIG. 6 is a diagram showing an example of the contents of the database stored in the color statistic storage unit 15. In this database DB, persons are classified into six object types 100 (home team players other than goalkeepers, home team side goalkeepers, away team players other than goalkeepers, away team side goalkeepers, referees, etc.), and identifiers 1 to 6 Is attached. Further, in this database DB, an average color vector 101 of players and a color covariance matrix 102 are stored in association with the object type 100 as color statistics. The average color vector 101 and the color covariance matrix 102 are obtained by measuring in advance the color of the player's uniform (clothing color) or the color of the player's whole body.
Returning to FIG. 2, the description will be continued.

The classification unit 16 assigns players to each team based on the color statistic for each identification number measured by the color statistic measurement unit 14 and the color statistic stored in the color statistic storage unit 15. Classify. Here, the classification unit 16 searches the color statistic storage unit 15 for the object type closest to the color statistic measured by the color statistic measurement unit 14, and sets the corresponding object type together with the identification number to the offside line. Output to the position output means 20.
For example, the color statistic measured by the color statistic measuring unit 14 is μ, and the average color vector and color covariance matrix which are the color statistic stored in the color statistic storage unit 15 are (μ _n , Σ _n ) (n
= 1, 2,...), The statistical distance C can be calculated using the Mahalanobis distance as shown in the equation (27).

Here, the classification means 16 outputs an offside line output indicating the timing of detecting the offside line to the offside line position output means 20 when the classification operation for each unit of video (for example, frame) is completed. Instructions shall be notified.
As described above, the player position detecting means 10 can recognize and output the three-dimensional coordinates (three-dimensional estimated player coordinates) of each player and the object type of the player from the input video.
Here, the offside line position output means 20 is added to the player position detection means 10 to constitute the offside line detection device 1 that detects the position of the offside line based on the position of the player.

The offside line position output means 20 is based on the object type for each player classified by the classification means 16 (type information such as the team to which the team belongs) and the three-dimensional estimated player coordinates estimated by the three-dimensional player coordinate estimation means 12. The position of the offside line is specified (estimated) and output.
For example, the offside line position output means 20 searches for the position of the second player counted from the rear side of the defensive side of the home team or away team of the soccer game in the three-dimensional estimated player coordinates.

Then, when paying attention to the home team, the offside line position output means 20 has an object type (object identifier) of “1 (home non-goal keeper)” indicating a player of the home team other than the goal keeper, Only the player with the identification number classified as “2 (home goalkeeper)” (see FIG. 6) indicating the goalkeeper is searched, and the position of the second player is searched from the rear side of the defensive side.
Further, when paying attention to the away team, the offside line position output means 20 has “3 (away away non-goalkeeper)” whose object type (object identifier) indicates an away team player other than the goalkeeper, Only the player with the identification number classified as “4 (away goalkeeper)” (see FIG. 6) indicating the goalkeeper is searched, and the position of the second player is searched from the back side of the defensive side.
In addition, in order to obtain the number of players from the back side of the defensive side using the player's three-dimensional coordinates (three-dimensional estimated player coordinates), the size of the touch line direction component (x coordinate) of the three-dimensional coordinates is determined. That is, when the goal is in the positive x-axis direction, a position where the x coordinate is the second largest is searched. Conversely, when the goal is in the negative x-axis direction, the position where the x coordinate is the second smallest is searched. The position (coordinates) with respect to the touch line serving as a reference for the offside line is specified.

  Here, a detailed configuration of the offside line position output means 20 will be described with reference to FIG. 7 (refer to FIG. 2 as appropriate). FIG. 7 is a block diagram showing the configuration of the offside line position output means. Here, the offside line position output unit 20 includes a coordinate storage unit 21 and an offside line coordinate estimation unit 22.

The coordinate storage means 21 stores the three-dimensional estimated player coordinates detected by the player position detection means 10 and the object type in association with the identification number. For example, as shown in FIG. 9, the object type c _i and the three-dimensional estimated player coordinates (x coordinate x _i , y coordinate y _i , z coordinate z _i ) are stored in association with the identification number.

  The offside line coordinate estimation unit 22 estimates the coordinates of the offside line based on the three-dimensional estimated player coordinates stored in the coordinate storage unit 21 and the object type. Here, the offside line coordinate estimation means 22 includes a coordinate search means 22a and a coordinate specification means 22b.

The coordinate search means 22a searches the corresponding coordinates from the coordinate storage means 21 when notified of the object type from the coordinate specifying means 22b described later. In addition to the object type, the coordinate search means 22a searches only the relevant coordinates from the coordinate storage means 21 when notified of the order of the size of a specific coordinate component (for example, x coordinate). The coordinates searched by the coordinate search means 22a are output to the coordinate specifying means 22b.
Note that the coordinate search means 22a does not search all applicable coordinates at the stage requested by the coordinate specification means 22b, but stores, for example, data for one frame (three-dimensional estimated player coordinates) in the coordinate storage means 21. In addition, at the stage where the object type) is stored, the data may be functioned as a coordinate sorting unit that rearranges the data for each object type in ascending or descending order of the size of the specific coordinate component (for example, x coordinate). Thus, for example, the contents of the coordinate storage means 21 shown in FIG. 9 are changed in the order of the size of the x coordinate (here, as shown in FIGS. 10A to 10E) for each object type. The data structure is arranged in descending order.

The coordinate specifying unit 22b specifies (estimates) the offside line based on the player or the positional relationship between the player and the referee. Here, it is assumed that the coordinate specifying unit 22b is activated when an offside line output instruction is notified from the classification unit 16. After the activation, the coordinate specifying unit 22b requests the coordinate search unit 22a for the position coordinates of the player or the referee based on the object type, and specifies the offside line by acquiring the position coordinates. The description of the method of identifying the offside line based on the position relationship between the player or the player and the referee will be made in the description of the operation of the offside line detection device 1.
Incidentally, the offside line position output unit 20, as shown in FIG. 8, by adding a filter means 23, may be configured as offside line position output unit 20 _2.

The filter unit 23 determines whether or not a person (player, referee) of an object type is valid as the existence position in the three-dimensional estimated player coordinates detected by the player position detection unit 10. The filter means 23 stores the three-dimensional estimated player coordinates and the object type in the coordinate storage means 21 in association with the identification number only when it is determined to be appropriate.
For example, the filter unit 23 determines whether or not the input data is valid by providing the following various determination conditions.

(Judgment condition example 1)
In general, in a soccer competition, the lineman is located in the vicinity of the offside line. That is, it can be said that a referee who is effective in estimating an offside line is a line referee. Therefore, the filter means 23 is only when the input object type c is “5” (see FIG. 6) indicating a referee, and the referee exists outside the predetermined range (for example, the following (28) (If the y-coordinate is less than 33 meters or exceeds 35 meters, as indicated by the formula), it is determined that the referee is not a lineman and the input data is discarded (not stored in the coordinate storage means 21).

(Judgment condition example 2)
In general, the goalkeeper often exists near the goal line. Therefore, the filter means 24 exists at a position where the goal keeper is separated from the goal line by a predetermined distance only when the input object type c is “2” or “4” (see FIG. 6) indicating the goal keeper. (For example, as shown in the following equation (29), when the x coordinate of each goalkeeper exceeds 30 m from the own goal), it is determined that the coordinate of the goalkeeper is incorrect, and the input data is Discard (do not store in coordinate storage means 21).

Thus, the offside line position output unit 20 _2, by providing the filter unit 23, it is possible to improve the estimation accuracy of the offside line.
As described above, the offside line detection device 1 can detect the coordinates of the offside line by detecting the player and the position of the referee for each team from the video (input video) taken of the soccer game. . Since the coordinates of the offside line detected in this way are coordinates in the world coordinate system, it becomes possible to visualize the offside line of the gamer competition on the screen by the conventional CG synthesis technique.

The configuration of the offside line detection device 1 has been described above, but the present invention is not limited to this. For example, the offside line detection device 1 can be realized by causing a general computer to execute a program and operating an arithmetic device or a storage device in the computer. This program (offside line detection program) can be distributed via a communication line, or can be distributed by writing on a recording medium such as a CD-ROM.
Here, the area determination unit 12b is configured to output only region information corresponding to the size of the player on a predetermined two-dimensional coordinate, but appears on the video instead of the area determination unit 12b. It is also possible to incorporate an adaptive area determination means that changes the area for determination in accordance with the size of the player on the three-dimensional coordinates.

  Here, the adaptive area determination means will be described with reference to FIG. 11 (refer to FIG. 2 as appropriate for other configurations). FIG. 11 is a block diagram showing a detailed configuration of the adaptive area determination means. Here, the adaptive area determination unit 17 includes a three-dimensional coordinate conversion unit 17a, a two-dimensional player region setting unit 17b, an area threshold generation unit 17c, and a variable area determination unit 17d.

  Based on the position and focal length of the camera, the three-dimensional coordinate conversion means (back projection conversion means) 17a converts the center-of-gravity coordinates, which are two-dimensional coordinates for each player area included in the area information generated by the labeling means 12a, into the world It is converted into three-dimensional coordinates in the coordinate system. The three-dimensional coordinate conversion unit 17a can be realized as the same as the backprojection conversion unit 12c described in FIG. That is, the center-of-gravity coordinates are converted into three-dimensional coordinates according to the equation (3). The three-dimensional coordinates converted and calculated here are output to the two-dimensional player region setting means 17b. Further, when “invalid” is obtained in the expression (3), the fact is notified to the two-dimensional player area setting means 17b.

  The two-dimensional player area setting means (projection conversion means) 17b is based on the three-dimensional coordinates converted by the three-dimensional coordinate conversion means 17a, the position and focal length of the camera, and a predetermined player size. A two-dimensional player area in which players are present is set. The two-dimensional player area setting means 17b can be realized as the same as the two-dimensional player area setting means 13 described in FIG. That is, the two-dimensional player area setting means 17b approximates the player's area based on the three-dimensional coordinates, projects the cylinder onto the two-dimensional coordinates, and includes a bounding box (25) described above. Is set as the player's two-dimensional player area. The two-dimensional player area (bounding box) set here is output to the area threshold generation means 17c. When the “invalid” is notified from the three-dimensional coordinate conversion unit 17a, the two-dimensional player area setting unit 17b notifies the area threshold generation unit 17c of “invalid” in association with the label number. .

The area threshold value generation means 17c generates a threshold value for determining whether or not the player area is valid as a player area based on the size of the bounding box set by the two-dimensional player area setting means 17b. This area threshold value generation means 17c calculates the threshold value which shows the effective range of the player area used by the variable area determination means 17d as the lower limit value A _min and the upper limit value A _max . For example, by setting the area of the bounding box D as D _S and the predetermined constants ρ _min and ρ _max as the area D _S , the lower limit value A _min and the upper limit value A _max are calculated as shown in the equation (30). To do.

  When the “invalid” is notified from the two-dimensional player region setting unit 17b, the area threshold generation unit 17c notifies the variable area determining unit 17d of “invalid” in association with the label number.

The variable area determination unit 17d determines whether or not the player region in the silhouette video is valid as the player region based on the threshold value generated by the area threshold value generation unit 17c. This variable area determination means 17d corresponds to the label number m (1 ≦ m ≦ M) with the lower limit value A _min and the upper limit value A _max indicating the effective range of the player area generated by the area threshold value generation means 17c as threshold values. area a _m of area information that is, the closed interval [a _min, a _max] to determine if included _{in, a m ∈ [a min,} a max] only if they meet, and outputs the area information. Here, the area information that satisfies the condition is output to the backprojection conversion means 12c. Note that the variable area determination unit 17d does not output region information corresponding to the label number when “invalid” is notified from the area threshold generation unit 17c.

As described above, by using the area determination unit 12b as the adaptive area determination unit 17, it is possible to determine whether the player area in the silhouette video is valid based on the size of the player in the three-dimensional coordinates. . As a result, even if the size of the player shown on the video changes due to the movement of the player, it can be determined whether or not it is the player's area, and the player's area can be specified more accurately.
When the adaptive area determination unit 17 is used instead of the area determination unit 12b, the back projection conversion unit 12c may be omitted and the output of the three-dimensional coordinate conversion unit 17a may be reused. That is, the variable area determining unit 17d uses the three-dimensional coordinates output by the three-dimensional coordinate conversion unit 17a as the label number corresponding to the region information satisfying A _m ∈ [A _min , A _max ]. At the same time, it is output to the detection / tracking means 12d as three-dimensional player coordinates.

[Operation of offside line detector]
Next, the operation of the offside line detection device will be described with reference to FIG. FIG. 12 is a flowchart showing the operation of the offside line detection device according to the first embodiment of the present invention. The flowchart of FIG. 12 shows the operation for one frame in the input video, and this operation is repeatedly executed for each frame that is sequentially input.

(Silhouette video generation step)
First, the offside line detection device 1 generates a silhouette video by dividing the frame into a player area and a background area for each input frame by the silhouette video generation means 11 (step S1). Here, the silhouette video generation means 11 binarizes the frame with the player area having the pixel value “1” and the background area having the pixel value “0”.

(3D player coordinate estimation step)
Then, the offside line detection device 1 assigns a label number to the player region (region of pixel value “1”) of the silhouette video by the labeling means 12a (step S2). The labeling means 12a calculates the area and center-of-gravity coordinates of the player area (single connection area) when performing the labeling process.
Then, the offside line detection device 1 has the area of the player area within the area corresponding to the size of the player determined in advance among the player areas to which the label number is given by the area determination unit 12b. Only the region is filtered (step S3). Thereby, an area not corresponding to the size of the player can be deleted.

Further, the offside line detection device 1 calculates the three-dimensional player coordinates by backprojecting the barycentric coordinates for each player area filtered in step S3 by the backprojection conversion means 12c (step S4). Here, the barycentric coordinates are two-dimensional coordinates, and the calculated three-dimensional player coordinates are three-dimensional coordinates in the world coordinate system.
Then, the offside line detection device 1 uses the detection / tracking unit 12d to calculate the three-dimensional player coordinates calculated in step S4 and the three-dimensional predicted player coordinates delayed one unit time (one frame) in step S9 described later. Are calculated for all combinations (step S5), and an identification number is assigned to the three-dimensional player coordinates based on the distance (step S6).

In other words, the detection / tracking means 12d determines the identification number associated with the shortest three-dimensional predicted player coordinate whose distance calculated in step S5 is equal to or less than a predetermined threshold and the three-dimensional player coordinate. To set. The detection / tracking unit 12d gives a new identification number to the three-dimensional player coordinates for which no identification number is set.
Thereby, the detection / tracking means 12d can track the same player area (player area).

Then, the offside line detection device 1 uses the prediction / estimation unit 12e to smooth the three-dimensional player coordinates in the time direction to estimate the three-dimensional estimated player coordinates, and also the player region after one unit time (one frame) Are predicted as three-dimensional predicted player coordinates (step S8). Here, by making the prediction / estimation means 12e function as a Kalman filter, the offside line detection device 1 can smoothly track the player.
Then, the offside line detection device 1 delays the three-dimensional predicted player coordinates predicted in step S8 by one unit time by the delay unit 12f (step S9). The three-dimensional predicted player coordinates delayed here are used in step S5 in the operation after one frame.

(2D player area setting step)
Further, the offside line detection device 1 uses the two-dimensional player region setting means 13 to calculate the three-dimensional estimated player coordinates estimated in step S8 based on the position and focal length of the camera and a predetermined player size. A two-dimensional player area, which is a two-dimensional area where a player exists in the input video, is set (step S10).

(Color statistics measurement step)
Then, the offside line detection device 1 uses the color statistic measurement means 14 within the area common to the two-dimensional player area set in step S10 and the silhouette video player area generated by the silhouette video generation means 11. The color statistic of the input video is measured (step S11). Here, for example, as a color statistic, an average color vector obtained by averaging color vectors in a region is used.

(Classification step)
Further, the offside line detection device 1 uses the classification unit 16 based on the color statistics for each identification number measured in step S11 and the color statistics stored in the color statistics storage unit 15. The players are classified for each team (step S12).
Thereby, the classification means 16 can classify to which object type the player reflected in the input video belongs, and can distinguish between the player on the home team side of the soccer competition and the player on the away team side. it can.

(Offside line position output step)
Then, the offside line detection device 1 uses the offside line position output unit 20 to classify the object type (such as a team) for each player classified by the classification unit 16 and the three-dimensional estimation estimated by the three-dimensional player coordinate estimation unit 12. Based on the player coordinates, the position of the offside line is estimated and output (step S13).

<Operation of offside line position output means>
Here, with reference to FIG. 13 and FIG. 14 (refer to FIG. 7 as appropriate), the operation of step S13 in FIG. 12 (the operation of the offside line position output means 20) will be specifically described. FIG. 13 is a flowchart showing an operation of estimating an offside line based on the positions of players (goalkeepers and non-goalkeepers) of the home team. FIG. 14 is a flowchart showing an operation of estimating an offside line based on the position of the player of the home team and the referee. In FIG. 13 and FIG. 14, “max {...}” is a function for obtaining the maximum value in the set {...}, and “max2 {...}” is the second largest value in the set {...}. Is a function for obtaining. Further, when there are a plurality of elements having the maximum value in the set {...}, it is assumed that max {...} = max2 {...} = “maximum value in the set {...}”.
Here, the three-dimensional estimated player coordinates estimated in step S8 and the object type classified in step S12 are associated with identification numbers, respectively, and the coordinate storage means 21 of the offside line position output means 20 is used. It is assumed that it is stored in

(Operation of offside line position output means [estimated by home player])
First, referring to FIG. 13 (refer to FIG. 7 as appropriate), the operation of estimating the position of the offside line based on the position of the player of the home team in the offside line position output means 20 will be described.
First, the offside line position output means 20 searches for whether the coordinates of the home team goalkeeper (home goalkeeper) are stored in the coordinate storage means 21, so that the number of home goalkeepers is "0". It is determined whether or not (step S20). Here, the coordinate specifying unit 22b of the offside line coordinate estimation unit 22 notifies the coordinate search unit 22a of the object type (here, the value “2” [home side goalkeeper]), and the coordinates of the home side goalkeeper coordinates. Instruct the search. Then, the coordinate search means 22a retrieves the coordinates of the home goal keeper from the coordinate storage means 21, and notifies the coordinate identification means 22b of the result. That is, the coordinate specifying unit 22b determines whether the number of home goalkeepers is “0” based on the search result from the coordinate search unit 22a.

Here, when the number of home goalkeepers is “0” (Yes in step S20), the coordinate specifying means 22b has the largest x-coordinate at the position of the home-side non-goalkeeper player who is a player other than the goalkeeper of the home team. A value is substituted into a variable _Xd (step S21). That is, the coordinate specifying unit 22b makes an object type (here, the value “1” [home-side non-goalkeeper]) and the coordinate size order (here, the first [maximum] Value]) and instruct the search. Then, the coordinate search means 22a searches the coordinate storage means 21 for the largest x-coordinate value at the position of the home-side non-goalkeeper and notifies the coordinate specifying means 22b of the result. Thereby, the coordinate specifying means 22b can obtain the largest x coordinate value at the position of the home non-goalkeeper. Then, the process proceeds to step S24.

On the other hand, when the number of home goalkeepers is not “0” (No in step S20), the coordinate specifying unit 22b is the most at the position of the home non-goalkeeper player (object type “1”) with respect to the coordinate search unit 22a. A search for the largest x-coordinate, the second largest x-coordinate, and the largest x-coordinate at the position of the home goalkeeper (object type “2”) is requested, and the respective values are set as variables X _d1 , X _d2 and X _g. (Step S22). When there are a plurality of the largest x coordinates, the “largest x coordinate” and the “second largest x coordinate” both indicate the same value (maximum value).

Then, the coordinate specifying means 22b substitutes the second largest value among the variables X _d1 , X _d2 and X _g for the variable X _d (step S23). When the second largest x-coordinate of the home-side non-goalkeeper, that is, the value of the variable X _d2 does not exist, the smaller value of the variables X _d1 and X _g is substituted into the variable X _d . At this stage, the coordinate specifying unit 22b estimates the variable _Xd as the offside line position.
Then, the coordinate specifying unit 22b outputs the value of the variable _Xd as the offside line position (x coordinate) (step S24).

(Operation of offside line position output means [home side player + estimation by referee])
Next, referring to FIG. 14 (refer to FIG. 7 as appropriate), the operation of estimating the position of the offside line based on the position of the player of the home team and the referee in the offside line position output means 20 will be described.
First, the offside line position output means 20 estimates the position of the offside line based on the position of the home player (step SA1). Note that step SA1 is the same as the operation of step S20 to step S23 described in FIG.

  Then, the offside line position output unit 20 determines whether or not the number of referees is “0” by searching whether or not the coordinates of the referee are stored in the coordinate storage unit 21 (step S30). Here, the coordinate specifying unit 22b of the offside line coordinate estimation unit 22 notifies the coordinate search unit 22a of the object type (here, the value “5” [referee]) and instructs the search of the referee's coordinates. . Then, the coordinate search means 22a searches the coordinates of the referee from the coordinate storage means 21, and notifies the result to the coordinate specification means 22b. That is, the coordinate specifying unit 22b determines whether the number of referees is “0” based on the search result from the coordinate search unit 22a.

Here, if the number of judgment is "0" (Yes in step S30), the coordinate specifying means 22b is substituted x-coordinate of the offside line position estimated in step SA1 (the value of variable X _d) for the variable X _p (Step S31). At this stage, since the referee does not exist on the video, the coordinate specifying unit 22b uses the position of the offside line estimated from the position of the home player, which is estimated in step SA1. Then, the process proceeds to step S35.

On the other hand, if the number of referees is not “0” (No in step S30), the coordinate specifying means 22b regards the referee with the largest x-coordinate value at the referee position as the line referee on the home-side offside line. substituting coordinates the variable X _r (step S32). That is, the coordinate specifying means 22b gives the object type (here, the value “5” [referee]) and the coordinate size order (here, the first [maximum value]) to the coordinate searching means 22a. Notify and direct search. Then, the coordinate search means 22a searches the coordinate storage means 21 for the largest x-coordinate value at the referee's position, and notifies the coordinate specifying means 22b of the result. Thereby, the coordinate specifying means 22b can obtain the largest x-coordinate value at the position of the referee.

Subsequently, the coordinate specifying means 22b obtains the largest x coordinate (variable _Xr ) at the referee position obtained in step S32 and the x coordinate (variable _Xd ) of the offside line estimated from the home player in step SA1. Compare. If the referee position is closer to the home goal line (goal post) than the home player position, the referee position is closer to the offside line position. . Therefore, here, the coordinate specifying unit 22b adds a predetermined threshold value θ (for example, 1 m) to the x coordinate (variable X _d ) of the position of the offside line estimated from the home player, and the x coordinate of the position of the referee (variable X _r) and comparing (step S33).

If the x coordinate (variable _Xr ) of the referee position is sufficiently larger than the x coordinate of the offside line position estimated in step SA1 (Yes in step S33), the coordinate specifying unit 22b sets the referee position offside. estimating the position of the line, it substitutes the value of variable X _r to the variable X _p (step S34). The coordinate specifying means 22b, in step _{S34, X p = (X d} + X r) / 2 of the by carrying out calculation, offside line position obtained from the position of the player and (X _d), the position of the referee The arithmetic mean value with the obtained off-side line position (X _r ) may be estimated as the off-side line position (X _p ).

On the other hand, when the x coordinate (variable _Xr ) of the referee position is not sufficiently larger than the x coordinate of the offside line position estimated in step SA1 (No in step S33), the operation of step S31 is executed.
The coordinate specifying means 22b outputs the value of the variable X _p as the offside line position (x-coordinate) (step S35).
Thus, the offside line position output means 20 can accurately estimate the position of the offside line by taking into account the position of the referee as well as the position of the player.
Here, the position of the off-side line on the home side is estimated. However, the smaller the x coordinate, the closer to the goal line on the away side, so by reversing the magnitude relationship of the coordinates, the off-side line on the away side It is also possible to estimate the position of.

  By the above operation, the offside line detection device 1 can sequentially output the position (coordinates) of the offside line that changes according to the movement of the player from the input video obtained by shooting the soccer game. As a result, the offside line position (coordinates) output from the offside line detection device 1 can be visualized on the screen in real time by using the conventional CG synthesis technology. Become.

<Second Embodiment>
[Configuration of offside line detector]
Next, with reference to FIG. 15, the structure of the offside line detection apparatus which concerns on 2nd embodiment of this invention is demonstrated. FIG. 15 is a block diagram showing a configuration of an offside line detection apparatus according to the second embodiment of the present invention. As shown in FIG. 15, the offside line detection apparatus 1B includes an object position detection unit 10B and an offside line position output unit 20B.

  The object position detection means 10B detects the position of the player, the referee, or the ball for each team in the input video from the input sports video (here, the input video obtained by shooting a soccer game). Here, the object position detection means 10B includes a silhouette video generation means 11, a 3D object coordinate estimation means 12B, a 2D player area setting means 13, a color statistic measurement means 14, a color statistic storage means 15, The classification means 16 is provided. The configuration other than the three-dimensional object coordinate estimation unit 12B is the same as the configuration of the player position detection unit 10 described with reference to FIG.

  The three-dimensional object coordinate estimating means 12B estimates the three-dimensional coordinates of each player object (including the referee) and the ball from the silhouette video generated by the silhouette video generating means 11. The three-dimensional object coordinate estimation means 12B is obtained by adding a function for estimating the three-dimensional coordinates of the ball to the three-dimensional player coordinate estimation means 12 described with reference to FIG. The estimation of the dimensional coordinates can be realized with the same configuration as the three-dimensional player coordinate estimation means 12. Here, only the area determination means 12Bb and the prediction / estimation means 12Be, which have different configurations by adding a function for estimating the three-dimensional coordinates of the ball to the three-dimensional player coordinate estimation means 12, will be described.

The area determination unit 12Bb functions as an area filter that determines whether the area in the region information generated by the labeling unit 12a is within a specific range, and outputs only the region information whose area is in the specific range. Is. Here, the lower limit value A _min and upper limit value A _{max of} the area corresponding to the size of the player (including the referee), and the lower limit value B _min and upper limit value B _{max of} the area corresponding to the size of the ball, It is set as a threshold value.

Then, the area determination unit 12Bb, the area A _m of the label number m (1 ≦ m ≦ M) (M is the number of simply connected region) region information corresponding to the found closed interval [A _min, A _max] or a closed interval [B _min, B _max] to determine if included _{in, a m ∈ [a min,} a max] or _{a m ∈ [B min, B} max] only if they meet, and outputs the area information. Here, the area information that satisfies the condition is output to the backprojection conversion means 12c. The area determination means 12Bb outputs person / ball determination information indicating whether the area is determined as a player (including a referee) or as a ball to the back projection conversion means 12c, and the information Is further notified to the detection / tracking means 12d and the prediction / estimation means 12Be in sequence.
The area determining means 12Bb calculates the circularity e as shown in the following equation (31), where S is the area of the ball region and l is the perimeter, and only the one closer to the circle is calculated. It may be selected as a region.

  Accordingly, the region H that approximates the size of the ball shown in FIG. 4 is not determined as a ball.

The prediction / estimation means 12Be smoothes the three-dimensional coordinates (three-dimensional object coordinates) of an object such as a player or a ball newly detected or tracked by the detection / tracking means 12d in the time direction. In addition to the estimated object coordinates, the three-dimensional coordinates of the object after one unit time are predicted as the three-dimensional predicted object coordinates.
The three-dimensional predicted object coordinates predicted by the prediction / estimation unit 12Be are output to the delay unit 12f together with an identification number that identifies the object. Further, the prediction / estimation means 12Be determines whether the three-dimensional estimated object coordinates are the player coordinates or the ball coordinates based on the person / ball determination information. The estimated object coordinates are output as the three-dimensional estimated player coordinates to the two-dimensional player area setting means 13 and the offside line position output means 20B. When the estimated object coordinates are the coordinates of the ball, the three-dimensional estimated object coordinates are set as the three-dimensional estimated ball coordinates. Output to the line position output means 20B.

Note that the prediction / estimation means 12Be outputs the identification number and the object type indicating that the object is a ball when outputting the three-dimensional estimated ball coordinates to the offside line position output means 20B. To do.
Thereby, the object position detection means 10B can detect the three-dimensional coordinates (three-dimensional estimated player coordinates) of an object such as a player (including a referee) and a ball from the input video.

The offside line position output means 20B estimates and outputs the position of the offside line based on the three-dimensional estimated player coordinates and the three-dimensional estimated ball coordinates estimated by the three-dimensional object coordinate estimation means 12B. Further, it is assumed that the player object type is notified from the classification unit 16, and the ball object type is notified from the prediction / estimation unit 12Be of the three-dimensional object coordinate estimation unit 12B.
The detailed configuration of the offside line position output unit 20B can be the same as that of the offside line position output unit 20 described with reference to FIGS. However, the coordinate storage means 21 of the offside line position output means 20B stores the position information (three-dimensional estimated ball coordinates) of the ball in addition to the position information of the player and the referee.

For example, as shown in FIG. 16A, the coordinate storage means 21 includes an object associated with an identification number including three-dimensional estimated ball coordinates (here, the object type = “7” coordinates). The type c _i and coordinates (x coordinate x _i , y coordinate y _i , z coordinate z _i ) are stored.
Further, the coordinate search unit 22a of the object coordinate estimation unit 22 rearranges the data stored in the coordinate storage unit 21 in ascending or descending order of the size of a specific coordinate component (for example, x coordinate) for each object type. It is also possible to function as coordinate sorting means. In this case, in addition to the object types shown in FIG. 14, the coordinate search means 22a converts the coordinates of the ball object type (here, “7”) into a specific coordinate component (for example, “7”) as shown in FIG. , X coordinates) may be arranged in ascending or descending order of magnitude and stored in the coordinate storage means 21.

[Operation of offside line detector]
Next, the operation of the offside line detection device 1B will be described. The operation of the object position detection means 10B is different from the player position detection operation described with reference to FIG. 12 only in that a ball detection is added. The operation of the output unit 20B will be specifically described.

(Operation of offside line position output means [home side player + referee + ball estimation])
Here, the operation of the offside line position output means 20B will be described with reference to FIG. 17 (refer to FIG. 7 as appropriate). FIG. 17 is a flowchart showing the operation of estimating the position of the offside line based on the position of the player, the referee and the ball of the home team in the offside line position output means.

First, the offside line position output unit 20B estimates the position of the offside line based on the positions of the home player and the referee (step SA2). Note that step SA2 is the same as the operations of step SA1 and steps S30 to S34 described in FIG.
Then, the offside line position output unit 20B determines whether or not the number of balls is “0” by searching whether or not the coordinates of the ball are stored in the coordinate storage unit 21 (step S40). Here, the coordinate specifying means 22b of the offside line coordinate estimation means 22 notifies the coordinate search means 22a of the object type (here, the value “7” [ball]) and instructs the search for the coordinates of the ball. . Then, the coordinate search means 22a searches for the coordinates of the ball from the coordinate storage means 21, and notifies the result to the coordinate specification means 22b. That is, the coordinate specifying unit 22b determines whether or not the number of balls is “0” based on the search result from the coordinate search unit 22a.

Here, when the number of balls is "0" (Yes in step S40), the coordinate specifying means 22b substitutes the x-coordinate of the offside line position estimated in step SA2 (the value of variable X _p) to the variable X (Step S41). At this stage, since the ball does not exist on the video, the coordinate specifying unit 22b uses the position of the offside line estimated from the position of the home player and the referee estimated in Step SA2. Then, the process proceeds to step S45.

On the other hand, when the number of balls is not “0” (No in step S40), the coordinate specifying means 22b substitutes the value of the largest x coordinate at the position of the ball into the variable _Xb (step S42). That is, the coordinate specifying unit 22b gives the object type (here, the value “7” [ball]) and the coordinate size order (here, the first [maximum value]) to the coordinate searching unit 22a. Notify and direct search. Then, the coordinate search means 22a searches the coordinate storage means 21 for the largest x coordinate value at the ball position, and notifies the coordinate specifying means 22b of the result. As a result, the coordinate specifying means 22b can obtain the largest x-coordinate value at the ball position.

Subsequently, the coordinate specifying means 22b has the largest x coordinate (variable _Xb ) at the ball position obtained in step S42, and the x coordinate of the offside line estimated from the home player and the referee in step SA2 (variable _Xp ). Are compared (step S43).
When the x coordinate (variable X _b ) of the ball position is larger than the x coordinate of the offside line position estimated in step SA2 (Yes in step S43), the coordinate specifying unit 22b determines the ball position as the offside line. And the value of the variable _Xb is substituted for the variable X (step S44).
On the other hand, when the x coordinate (variable X _b ) of the ball position is equal to or less than the x coordinate of the offside line position estimated in step SA2 (No in step S43), the operation of step S41 is executed.

Then, the coordinate specifying unit 22b outputs the value of the variable X as an offside line position (x coordinate) (step S45).
Thus, the offside line position output means 20B can estimate the position of the offside line more accurately by taking into account not only the position of the player and the referee but also the position of the ball.
Here, the position of the off-side line on the home side is estimated. However, the smaller the x coordinate, the closer to the goal line on the away side, so by reversing the magnitude relationship of the coordinates, the off-side line on the away side It is also possible to estimate the position of.

It is explanatory drawing for demonstrating the outline | summary of the offside line detection apparatus which concerns on this invention. It is the block diagram which showed the structure of the offside line detection apparatus which concerns on 1st embodiment of this invention. It is a figure which shows an example of the input image | video input into the offside line detection apparatus which concerns on this invention. It is a figure which shows the silhouette image | video which binarized the input image | video of FIG. It is a figure which shows the concept which provided the label number to the player area | region of the silhouette image | video of FIG. It is a figure which shows an example of the data of the database memorize | stored in the color statistic storage means. It is the block diagram which showed the structure of the offside line position output means. It is the block diagram which showed the other structure of the offside line position output means. It is a data structure figure which shows an example of the memory content of a coordinate memory | storage means. It is a data structure figure which shows an example of the memory content of the coordinate memory | storage means sorted for every object classification by the specific coordinate component. It is a block diagram which shows the structure of an adaptive area determination means. It is a flowchart which shows operation | movement of the off-side line detection apparatus which concerns on 1st embodiment of this invention. It is a flowchart which shows the operation | movement which an offside line position output means estimates an offside line based on the position of the player of a home team. It is a flowchart which shows the operation | movement which an offside line position output means estimates an offside line based on the player of a home team, and the position of a referee. It is the block diagram which showed the structure of the offside line detection apparatus which concerns on 2nd embodiment of this invention. The memory content of the coordinate memory | storage means of the off-side line detection apparatus which concerns on 2nd embodiment of this invention is shown, (a) is a data structure figure which shows an example of the memory content, (b) extracts only the coordinate of a ball | bowl. It is a data structure figure which shows the done example. It is a flowchart which shows the operation | movement which estimates the position of an offside line based on the position of the player of the home team, a referee, and a ball in an offside line position output means.

Explanation of symbols

1, 1B Offside line detection device 10 Player position detection means (player position detection device)
10B Object position detection means 11 Silhouette video generation means 12 3D player coordinate estimation means 12B 3D object coordinate estimation means 12a Labeling means 12b Area determination means 12c Backprojection conversion means 12d Detection / tracking means 12e Prediction / estimation means 12f Delay means 13 Two-dimensional player area setting means 14 Color statistic measurement means 15 Color statistic storage means 16 Classification means 17 Adaptive area determination means 17a Three-dimensional coordinate conversion means 17b Two-dimensional player area setting means (second two-dimensional player area setting means)
17c Area threshold generation means 17d Variable area determination means 20, 20B Offside line position output means 21 Coordinate storage means 22 Offside coordinate estimation means 22a Coordinate search means 22b Coordinate specification means 23 Filter means

Claims (12)

A player position detection device for detecting the position of a player for each team from an input video obtained by photographing a sports competition with a camera,
Silhouette video generation means for generating a silhouette video divided into a player area and a background area in time series from the input video,
Based on the position of the player area in the silhouette video generated by the silhouette video generation means, the position where the camera is installed, and the focal length at the time of shooting, the position of the player area is tracked in three-dimensional coordinates. 3D player coordinate estimation means for estimating the position of the player as 3D estimated player coordinates,
Based on the three-dimensional estimated player coordinates estimated by the three-dimensional player coordinate estimating means, the position and focal length of the camera, and a predetermined player size, the player exists in the input video 2 Two-dimensional player area setting means for setting a three-dimensional player area;
A color statistic measuring means for measuring a color statistic of the input video in a region common to the two-dimensional player region set by the two-dimensional player region setting unit and the player region in the silhouette video; ,
Color statistic storage means for storing at least a color statistic based on the clothing color of the team to which the player belongs;
Classification means for classifying the players by team based on the color statistics stored in the color statistics storage means and the color statistics measured by the color statistics measurement means;
A player position detecting device comprising: The three-dimensional player coordinate estimation means includes:
Based on a predetermined size of the player, the player region in the silhouette video comprises an area determination means for determining whether the player region is valid as the player region,
The player position detecting device according to claim 1, wherein the three-dimensional estimated player coordinates are estimated for a player region determined to be effective by the area determining means. The three-dimensional player coordinate estimation means includes:
3D coordinate conversion means for converting 2D coordinates for each player area in the silhouette video into 3D coordinates based on the position and focal length of the camera;
Based on the three-dimensional coordinates converted by the three-dimensional coordinate conversion means, the position and focal length of the camera, and a predetermined player size, a two-dimensional player region in which the player exists in the input video A second two-dimensional player area setting means for setting
Area threshold value generating means for generating a threshold value for determining whether or not the player area is valid as the player area based on the size of the two-dimensional player area set by the second two-dimensional player area setting means; ,
Based on the threshold generated by the area threshold generation means, variable area determination means for determining whether the player area in the silhouette video is valid as the area of the player,
The player position detecting device according to claim 1 or 2, wherein the three-dimensional estimated player coordinates are estimated for a player region determined to be effective by the variable area determining means. The three-dimensional player coordinate estimation means includes:
Based on the distance between the three-dimensional predicted player coordinates that are three-dimensional coordinates of the player area predicted in one unit time in the past and the three-dimensional player coordinates that are three-dimensional coordinates of the current player area, Detecting and tracking means for detecting and tracking for each player;
The three-dimensional player coordinates for each player detected and tracked by this detection / tracking means are smoothed in the time direction to obtain three-dimensional estimated player coordinates, and the three-dimensional coordinates of the player area after one unit time. Predicting / estimating means for predicting as the three-dimensional predicted player coordinates,
Delay means for delaying the three-dimensional predicted player coordinates predicted by the prediction / estimation means by one unit time;
The player position detecting device according to any one of claims 1 to 3, further comprising:   The statistic of the referee's clothing color is stored in the color statistic storage means, so that the classification means sets the referee as a classification target in addition to the player. The player position detection device according to claim 4. An offside line detection device that detects an offside line from an input image obtained by shooting a soccer game with a camera,
The player position detecting device according to any one of claims 1 to 4, wherein a player position is detected for each team from an input video obtained by photographing the soccer game as a sports game with a camera.
In this player position detection device, the position of the offside line is specified based on the team for each player classified by the classification means and the three-dimensional estimated player coordinates estimated by the three-dimensional player coordinate estimation means. Offside line position output means for outputting;
An off-side line detection device comprising: The offside line position output means includes
Coordinate storage means for storing the three-dimensional estimated player coordinates in association with type information for identifying the team;
Off-side line coordinate estimation means for estimating the position of the off-side line based on a specific coordinate component of the player divided into the team in the three-dimensional estimated player coordinates stored in the coordinate storage means;
The off-side line detection apparatus according to claim 6, comprising: An offside line detection device that detects an offside line from an input image obtained by shooting a soccer game with a camera,
The player position detection device according to claim 5, wherein at least positions of the player and the referee are detected from an input video obtained by photographing the soccer game with a camera as a sports game,
In this player position detection device, the position of the offside line is identified based on the players and referees by team classified by the classification means and the three-dimensional estimated player coordinates estimated by the three-dimensional player coordinate estimation means. Offside line position output means for outputting,
The offside line position output means includes
Coordinate storage means for storing the three-dimensional estimated player coordinates in association with the type information for identifying the player and the referee for each team,
Off-side line coordinate estimation means for estimating the position of the off-side line based on specific coordinate components of the players and the referees classified into the team in the three-dimensional estimated player coordinates stored in the coordinate storage means;
An off-side line detection device comprising:   8. A filter means for storing, in the coordinate storage means, only coordinates existing within a predetermined range among the three-dimensional estimated player coordinates detected by the player position detecting device. Or the off-side line detection apparatus of Claim 8. An offside line detection device that detects an offside line from an input image obtained by shooting a soccer game with a camera,
From the input video, silhouette video generation means for generating a silhouette video divided into an object area and a background area indicating a player or a ball in time series,
Based on the position and size of the object area in the silhouette video generated by the silhouette video generation means, the position where the camera is installed, and the focal length at the time of shooting, the position of the object area is expressed in three-dimensional coordinates. 3D object coordinate estimation means for estimating the position of the player and the ball as 3D estimated player coordinates and 3D estimated ball coordinates by tracking;
Based on the three-dimensional estimated player coordinates estimated by the three-dimensional object coordinate estimating means, the position and focal length of the camera, and a predetermined player size, the player exists in the input video 2 Two-dimensional player area setting means for setting a three-dimensional player area;
A color statistic measuring unit for measuring a color statistic of the input video in a region common to the two-dimensional player region set by the two-dimensional player region setting unit and the object region in the silhouette video; ,
Color statistic storage means for storing at least a color statistic based on the clothing color of the team to which the player belongs;
Classification means for classifying the players by team based on the color statistics stored in the color statistics storage means and the color statistics measured by the color statistics measurement means;
Offside line position output means for specifying and outputting the position of the offside line based on the team for each player classified by the classification means, and the three-dimensional estimated player coordinates and the three-dimensional estimated ball coordinates;
An off-side line detection device comprising: By storing the statistic of the referee's clothing color in the color statistic storage means, in addition to the player, the classification means takes the umpire as an object of classification,
The offside line position output means includes
Coordinate storage means for storing the three-dimensional estimated player coordinates and the three-dimensional estimated ball coordinates in association with type information for identifying the player, the referee and the ball;
Offside line coordinate estimation means for estimating the position of the offside line based on specific coordinate components of the player, the referee, and the ball divided into the team in the three-dimensional estimated player coordinates stored in the coordinate storage means When,
The off-side line detection device according to claim 10, comprising: In order to detect off-side lines from input images taken with soccer cameras,
Silhouette video generation means for generating a silhouette video divided into a player area and a background area in time series from the input video,
Based on the position of the player area in the silhouette video generated by the silhouette video generation means, the position where the camera is installed, and the focal length at the time of shooting, the position of the player area is tracked in three-dimensional coordinates. 3D player coordinate estimation means for estimating the position of the player as 3D estimated player coordinates,
Based on the three-dimensional estimated player coordinates estimated by the three-dimensional player coordinate estimating means, the position and focal length of the camera, and a predetermined player size, the player exists in the input video 2 2D player area setting means for setting a 3D player area,
A color statistic measuring means for measuring a statistic of the color of the input video in a region common to the two-dimensional player region set by the two-dimensional player region setting unit and the player region in the silhouette video;
The color statistic stored in the color statistic storage means storing at least the color statistic based on the clothing color of the team to which the player belongs, and the color statistic measured by the color statistic measurement means Classification means for classifying the players by team based on
Offside line position output means for specifying and outputting the position of the offside line based on the team for each player classified by the classification means and the three-dimensional estimated player coordinates estimated by the three-dimensional player coordinate estimation means ,
An offside line detection program characterized by functioning as

Images