JP5751996B2 - Subject three-dimensional region estimation method and program - Google Patents

Description

  The present invention relates to a method and a program for estimating a tertiary position of a subject, which is the most important process in the synthesis of a free viewpoint video.

  A method for synthesizing an arbitrary viewpoint image using only image information without using strong camera calibration as a target has been proposed. However, most of the conventional techniques for estimating a player's three-dimensional position with high accuracy for sports images performed in a large outdoor space such as soccer use only single camera information.

  Non-Patent Document 1 outlines a particle filter that is rapidly spreading to various application fields from the ease of implementation as a high-dimensional state vector estimation method in a general state space model.

Tomoyuki Higuchi: “Particle Filter”, IEICE Journal, Vol.88, No. 12, pp. 989-994 (2005)

  However, the conventional technique using only single camera information has a problem that the estimation accuracy is lowered due to the occurrence of an occlusion region where subjects overlap each other.

  In view of the above problems, the present invention integrates not only a single camera image but also a plurality of camera images taken synchronously, thereby improving the accuracy of the subject 3D position estimation. An object is to provide an estimation method and a program.

In order to achieve the above object, a method for estimating the three-dimensional world coordinates of a subject according to the present invention includes a three-dimensional world coordinate of a subject in an initial frame, a plurality of frames taken by a plurality of cameras, and a camera image including the subject. A method for estimating a three-dimensional world coordinate of a subject in a subsequent frame , the step of projecting the three-dimensional world coordinate of the subject in the initial frame to an XY coordinate on a specific plane; and on the specific plane in the subsequent frame estimating on the basis of XY coordinates of the object on the time axis information, the steps to evaluate the XY coordinates of an object on a specific plane, only including the step of projecting the XY coordinates on the specific plane, the The 3D world coordinates of the subject are specified as the bird's-eye view plane obtained when the field surface where the subject is present is seen from directly above. The step of projecting onto the XY coordinates on the overhead view plane as a plane, approximating the subject as a plurality of particles existing on the overhead view plane, and evaluating the XY coordinates of the subject is as follows: Likelihood indicating whether or not it seems to be a subject is calculated from each camera image in which the subject is approximated as the particle, and the subject on the overhead plane is calculated from the sum of the likelihood for each subject calculated from all camera images. The XY coordinates of the subject are evaluated for each subject, and the three-dimensional world coordinates of the subject in the subsequent frame are estimated by mapping with the XY coordinates evaluated to be the subject .

  In the step of estimating based on the time axis information, it is also preferable to formulate the moving direction on the bird's eye plane with respect to the particles forming the subject based on the speed based on the XY coordinate difference between frames.

The step of evaluating the XY coordinates of the subject includes a sub-step of estimating a plane projection matrix with respect to the overhead plane of each camera, and each camera image projected by the plane projection matrix of each particle forming the subject. a sub-step of calculating the likelihood, the likelihoods of the particles forming the subject, it is also preferred to include a sub-step of estimating the XY coordinates of the particles forming the subject with the subsequent frame.
Including

Further, the sub-step of estimating a plane projection matrix for the overhead plane of the camera gives four or more pixels observed in the camera image for feature points whose XY coordinates on the overhead plane are known, and a minimum of 2 It is also preferable to estimate a plane projection matrix with respect to the overhead view plane by multiplication.

The sub-step of calculating the likelihood of each particle projects the XY coordinates of the particles onto each camera image using a plane projection matrix with respect to the overhead view plane , and the XY coordinates of the particles are projected on each camera image. It is also preferable to set a rectangular area with the bottom pixel as the center of the base, calculate the subjectivity of each pixel included in the rectangular area, and calculate the likelihood of the particle from the average of the subjectness.

Further, the subjectivity of each pixel is calculated from the likelihood based on the dynamic background model constructed immediately before the frame at the current time, information based on the uniform color of the subject, and information based on the background pattern color. It is also preferable.

Also, sub-step of estimating the XY coordinates of the particles forming the subject with the subsequent frame, extinguished particles below a certain likelihood, the larger particles near than a certain likelihood, to reposition the new grain processing It is also preferable to contain .

A program characterized by estimating a three-dimensional world coordinates of an object according to the present invention for achieving the above object, a three-dimensional world coordinates of the object in the initial frame, a plurality of frames taken by a plurality of cameras, the subject A computer for estimating the three-dimensional world coordinates of the subject in the subsequent frame from the camera image including the image, and means for projecting the three-dimensional world coordinates of the subject in the initial frame to XY coordinates on a specific plane; and the subsequent frame said means for estimating, based the XY coordinates of an object on a particular plane in the time axis information, to function as a commentary worth means XY coordinates of the object on the particular plane, means for projecting the XY coordinates on the particular plane in Is obtained when the three-dimensional world coordinates of the subject are viewed from directly above the field surface on which the subject exists. Projecting onto the XY coordinates on the overhead plane using the viewpoint plane as the specific plane, approximating the subject as a plurality of particles existing on the overhead plane, and evaluating the XY coordinates of the subject on the overhead plane A likelihood indicating whether or not the XY coordinates of the subject are likely to be a subject is calculated from each camera image in which the subject is approximated as the particle, and the overhead is calculated from the sum of the likelihood for each subject calculated from all camera images. Whether the XY coordinates of the subject on the plane are likely to be the subject is evaluated for each subject, and the three-dimensional world coordinates of the subject in the subsequent frame are mapped and estimated with the XY coordinates evaluated as likely to be the subject. Estimate the 3D world coordinates of the subject.

  According to the present invention, it is possible to greatly improve the estimation accuracy of the subject three-dimensional region and improve the composite image quality of the free viewpoint video.

2 shows a flowchart according to the invention. A rectangular area in a camera image is shown. The camera image and subject XY coordinate map image in an initial frame are shown. A camera image and a subject XY coordinate map image in the frame 30 are shown.

  The best mode for carrying out the present invention will be described in detail below with reference to the drawings. The proposed method is characterized by handling time axis information and information between multiple cameras in an integrated manner. Specifically, in a specific frame, map with the XY coordinates obtained when the three-dimensional position of each player is looked down from directly above, and in the subsequent frame, motion prediction based on inter-frame information of the map, and between cameras A map is automatically generated by evaluating the player's characteristics (projecting candidate points to the camera viewpoint and evaluating whether or not the player is a player) in consideration of the correspondence in Hereinafter, description will be given based on this flowchart. In this embodiment, it is assumed that the three-dimensional position of the non-fixed zoom camera does not change.

  Start: Input multi-view video for multiple frames. That is, camera images of a plurality of frames taken by two or more cameras are input.

  Step 1: Estimate a planar projection matrix for the field plane of each camera. For example, the plane projection matrix Hc can be estimated by giving a plurality of pixel coordinates in the camera image for feature points whose three-dimensional coordinates are known in advance, such as a line on the field (reference: Japanese Patent Application No. 2010-032136). ). It is also possible to perform a strong camera calibration to estimate the projection relationship between the Euclidean coordinate system in the space and the camera coordinate system in advance, and to obtain the planar projection matrix Hc in advance.

The planar projection matrix Hc is a 3 × 3 matrix, and projects the point (u, v) in the two-dimensional coordinates of the target field to the camera pixel coordinates (u ′, v ′). That means
s (u ′, v ′, 1) ^T = Hc (u, v, 1) ^T
It is. Note that s is a scalar for normalization.

Step 2: Approximate each subject with a plurality of particles. The three-dimensional world coordinates are projected onto XY coordinates on the bird's-eye view plane obtained when the field surface on which each subject exists is viewed from directly above, and each subject is approximated as a plurality of particles existing on the bird's-eye view plane. The subject is composed of N particles, and the state of the subject at time t is formulated as a large number of discrete particle groups. The subject at time t is
Of N vectors. If there are M subjects in the camera image, all subjects in the camera image are approximated by a plurality of particles with N × M vectors. As the initial value of this vector, at least two frames (t = 0, 1) are input in advance, and in the following steps, the positions of the particles forming the subject in the subsequent frames are estimated, and the XY of the subject is determined. Estimate the position in coordinates.

  Step 3: Formulate a particle state transition model. For the particles forming each subject, the moving direction on the overhead view plane is formulated by the velocity (U, V) based on the XY coordinate difference between frames.

The movement vector (U _t , V _t ) representing the velocity of the particles at frame t is
(U _t , V _t ) = (X _t −X _t−1 , Y _t −Y _t−1 )
Is formulated.

When the position of the particle for two frames (t = 0, 1) is input according to this equation, (U ₁ , V ₁ ) is obtained, and the position of the particle at t = 2 (U ₂ ) is obtained using this velocity. , V ₂ ). The position of the particles can be obtained in the same manner for t = 3 or more.

Step 4: Calculate the likelihood of particles. Let t be the frame for estimating the position of the subject in the XY coordinates. From the XY coordinates (X ₀ ⁽ⁿ⁾ , Y ₀ ⁽ⁿ⁾ ) of the particles in the initial frame (t = 0), the XY coordinates (X _t ⁽ⁿ⁾ , Y _t of the particles in the frame t are obtained from the equation of Step 3. ^(N) ) is obtained. The obtained (X _t ⁽ⁿ⁾ , Y _t ⁽ⁿ⁾ ) is projected onto the pixel (x _{t, c} ⁽ⁿ⁾ , y _{t, c} ⁽ⁿ⁾ ) in each camera image by the plane projection matrix Hc.
s (x _{t, c} ⁽ⁿ⁾ , y _{t, c} ⁽ⁿ⁾ , 1) ^T = Hc (X _t ⁽ⁿ⁾ , Y _t ⁽ⁿ⁾ , 1) ^T
Note that n = 1,..., N, and c indicates a camera. For example, when there are C cameras, c = 1,.

Next, in the camera image, a rectangular region having the pixel (x _{t, c} ⁽ⁿ⁾ , y _{t, c} ⁽ⁿ⁾ ) as the center point of the base is set. For example, assuming that the width of the rectangle is w and the height is v, for the pixel xy, (x−w / 2, y), (x + w / 2, y), (x−w / 2, y + v), (x + w) A rectangular area of 4 points of / 2, y + v) is set. FIG. 2 shows a rectangular area set for the pixel xy.

Next, for each pixel included in the rectangular area, the likelihood indicating the subjectness is calculated, and the average value of the calculated likelihood is defined as the particle likelihood (Φ (xy)) at the pixel xy. Φ (xy) is
Φ (xy) = (1 / number of cameras) × (1 / number of pixels in rectangle) ΣΣφc (x ′, y ′)
It can be expressed.
Here, φc (x ′, y ′) is a likelihood indicating the subjectness at the pixel (x ′, y ′) in the rectangular area in the camera image by the camera c, and the first Σ is for all the cameras. The next Σ is the sum for all pixels in the rectangle.

  Note that the likelihood indicating the likelihood of the subject in each pixel is calculated based on the dynamic background model built up to time t-1. The dynamic background model is formulated by assuming a Gaussian distribution based on the average and variance of a plurality of frames for each pixel.

  Furthermore, when the color of the uniform of the person who is the subject is known in advance, the color of the subject in the rectangular area is compared with the color of the uniform, and the difference information can be added to the likelihood indicating the likelihood of the subject. Further, when the background color is known in advance, the background color of the rectangular area is compared with the color of the background, and the difference information can be added to the likelihood indicating the subjectness.

Step 5: Rearrange in the vicinity of particles having a certain likelihood or more. In the above step, the likelihood of the particle is calculated for the pixel (x _t ⁽ⁿ⁾ , y _t ⁽ⁿ⁾ ). Here, particles having a likelihood less than or equal to a certain amount are eliminated, and the same number of particles as the number of particles that have been eliminated are rearranged around particles having a likelihood that is greater than a certain value. For example, in the pixel (x _t ⁽ⁿ⁾ , y _t ⁽ⁿ⁾ ), when the likelihood of n = 1 to 100 is below a certain value, (x _t ⁽¹⁾ , y _t ⁽¹⁾ ) to (x _t ^{( 1) 100)} , y _t ⁽¹⁰⁰⁾ ) particles are extinguished and new (x _t ^(N) , y _t ^(N) ) in the vicinity of (x _t ⁽¹⁰¹⁾ , y _t ⁽¹⁰¹⁾ ) ( x _{t ^(1),} from ^{_{y t (1)) (x}} t (100), a new particle of ^{y t (100))} are rearranged.

  As described above, a plurality of particles that approximate the subject are estimated in the frame t, and the XY coordinates of the subject in the frame t on the overhead view plane are estimated from the plurality of particles.

  Next, examples of the present invention will be described. FIG. 3 shows a camera image and a subject XY coordinate map image in the initial frame. 3A shows an image of the camera 1, FIG. 3B shows an image of the camera 2, and FIG. 3C shows an object XY coordinate map image. Each point in FIG. 3C indicates a subject (a player in this embodiment). The XY coordinates of a plurality of particles constituting the subject are input as initial values.

  FIG. 4 shows a camera image and a subject XY coordinate map image in the frame 30. 4A shows an image of the camera 1, FIG. 4B shows an image of the camera 2, and FIG. 4C shows a subject XY coordinate map image. Each shows an image at t = 30. Since the present embodiment has 30 frames per second, the camera image in FIG. 4 is an image one second after the camera image in FIG. FIG. 4C is an image of a subject particle estimated in the frame 30 by the method of the present invention.

  As described above, the present invention maps the XY coordinates obtained when looking down on the three-dimensional position of each player from directly above in a specific frame, and performs motion prediction based on the inter-frame information of the map in subsequent frames. Then, the subject area is estimated by evaluating the player likeness considering the correspondence between the cameras.

  Moreover, all the embodiments described above are illustrative of the present invention and are not intended to limit the present invention, and the present invention can be implemented in other various modifications and changes. Therefore, the scope of the present invention is defined only by the claims and their equivalents.

Claims (8)

A method for estimating a three-dimensional world coordinate of a subject in a subsequent frame from a three-dimensional world coordinate of the subject in an initial frame and a camera image including the subject of a plurality of frames taken by a plurality of cameras,
Projecting the three-dimensional world coordinates of the subject in the initial frame to XY coordinates on a specific plane;
Estimating XY coordinates of the subject on the specific plane in the subsequent frame based on time axis information;
A step of evaluating the XY coordinates of the object on the particular plane,
Only including,
The step of projecting to the XY coordinates on the specific plane includes the three-dimensional world coordinates of the subject on the bird's-eye view plane with the bird's-eye view plane obtained when the field surface on which the subject exists is viewed from directly above as the specific plane XY coordinates of the object, approximating the subject as a plurality of particles present on the overhead view plane,
The step of evaluating the XY coordinates of the subject calculates a likelihood indicating whether the XY coordinates of the subject on the overhead view plane are likely to be a subject from each camera image in which the subject is approximated as the particles, From the sum of the likelihood for each subject calculated from the above, it is evaluated for each subject whether or not the XY coordinates of the subject on the overhead view plane are those of the subject, and the three-dimensional world coordinates of the subject in the subsequent frame are A method of estimating the three-dimensional world coordinates of a subject, characterized by mapping and estimating with XY coordinates evaluated as likely to be the subject. The step of estimating based on the time axis information includes:
The method according to claim 1 , wherein the moving direction on the bird's eye plane is formulated by a speed based on an XY coordinate difference between frames for the particles forming the subject. The step of evaluating the XY coordinates of the subject includes
A sub-step of estimating a plane projection matrix for the overhead plane of each camera;
A sub-step of calculating the likelihood of each particle forming the subject from each camera image projected by the planar projection matrix ;
From the likelihood of each particle forming the subject, the sub-steps of estimating the XY coordinates of the particles forming the subject with the subsequent frame,
The method according to claim 1 or 2 , comprising: The sub-step of estimating a plane projection matrix for the overhead plane of the camera is as follows:
The feature point whose XY coordinates on the overhead plane are known has four or more pixels observed in the camera image, and a plane projection matrix for the overhead plane is estimated by a least square method. 3. The method according to 3 . The sub-step of calculating the likelihood of each particle includes
Projecting the XY coordinates of the particles onto each camera image by a plane projection matrix for the overhead plane,
In each of the camera images, a rectangular region whose center is the pixel on which the XY coordinates of the particles are projected is set.
Calculating the subjectivity of each pixel contained within the rectangular area;
The method according to claim 3 or 4 and calculates the likelihood of the particles from an average of the object ness. The subjectivity of each pixel is
Likelihood based on a dynamic background model built up to just before the current frame,
Information based on the subject's uniform color,
Information based on background pattern color,
The method according to claim 5 , wherein the method is calculated from: The sub-step of estimating the XY coordinates of the particle group forming the subject in the subsequent frame is as follows:
The method according to any one of claims 3 to 6 , further comprising a process of eliminating particles having a certain likelihood or less and rearranging new particles in the vicinity of the particles having a certain likelihood. A computer for estimating the three-dimensional world coordinates of the subject in the subsequent frame from the three-dimensional world coordinates of the subject in the initial frame and the camera images including the subject of the plurality of frames taken by a plurality of cameras.
Means for projecting the three-dimensional world coordinates of the subject in the initial frame to XY coordinates on a specific plane;
Means for estimating XY coordinates of the subject on the specific plane in the subsequent frame based on time axis information;
It means for evaluating the XY coordinates of the object on the particular plane,
To function ,
The means for projecting onto the XY coordinates on the specific plane has the three-dimensional world coordinates of the subject as the specific plane as an overhead plane obtained when the field plane on which the subject exists is viewed from directly above the overhead plane. XY coordinates of the object, approximating the subject as a plurality of particles present on the overhead view plane,
The means for evaluating the XY coordinates of the subject calculates a likelihood indicating whether the XY coordinates of the subject on the overhead view plane are likely to be a subject from each camera image in which the subject is approximated as the particle, From the sum of the likelihood for each subject calculated from the above, it is evaluated for each subject whether or not the XY coordinates of the subject on the overhead view plane are those of the subject, and the three-dimensional world coordinates of the subject in the subsequent frame are A program characterized by estimating a three-dimensional world coordinate of a subject to be estimated by mapping with XY coordinates evaluated as being likely to be the subject.