JP2019057070A - Image processing device, image processing method, and program - Google Patents

Abstract

To enable the position of a subject in a real space to be specifiable when designating the position of a viewpoint pertaining to a virtual viewpoint image.SOLUTION: An image processing device of the present invention comprises: acquisition means for acquiring, on the basis of an image in which imaging target areas of a plurality of cameras are captured, the position information of an object seen in the image; display control means for causing information that indicates the object to be displayed at a position, in a computer graphics indicating the imaging target areas, that corresponds to the position information acquired by the acquisition means; and acceptance means for accepting the designation of a position of viewpoint pertaining to a virtual viewpoint image while the information indicating the object is being displayed by the display control means.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an image processing technique for setting a virtual camera path when generating a virtual viewpoint image.

  Using multiple real camera images, reproduce the image taken from the camera virtually placed in the three-dimensional space, and not only the image taken from the actual camera installation position but also the image of any viewpoint There is a technique for generating a viewpoint image. Patent Document 1 discloses a technique for generating a virtual viewpoint image from images taken by a plurality of cameras.

JP 2008-217243 A

  When specifying the position of the viewpoint related to the virtual viewpoint image, if the position of the subject is not known, the user can determine which position is the position of the viewpoint related to the virtual viewpoint image and obtain the desired virtual viewpoint image. It was difficult to judge. Therefore, a technique for enabling the user to specify the position of the subject in real space is desired in order to easily specify the viewpoint related to the desired virtual viewpoint image.

  An object of the present invention is to provide a technique for enabling the position of a subject in a real space to be specified when designating the position of a viewpoint related to a virtual viewpoint image.

  An image processing apparatus according to an aspect of the present invention is an image processing apparatus that accepts designation of a viewpoint position related to a virtual viewpoint image generated using a plurality of images captured by a plurality of cameras. In an acquisition unit that acquires position information of an object shown in the image based on an image obtained by shooting a shooting target area of the camera, and corresponding to the position information acquired by the acquiring unit in a computer graphic indicating the shooting target area Display control means for displaying information indicating the object at a position, and receiving means for receiving designation of the position of the viewpoint related to the virtual viewpoint image in a state where the information indicating the object is displayed by the display control means; It is characterized by providing.

  An image processing apparatus according to another aspect of the present invention obtains two-dimensional position information of a subject from conversion means for converting an image obtained by photographing a field into a bird's eye view image obtained by viewing the field from the sky, and the bird's eye view image. First generation means for generating, using the two-dimensional position information, generating means for generating a CG scene in which the area of the field including the subject is reproduced by a CG (computer graphic) model, and using the CG scene Second acquisition means for acquiring the three-dimensional position information of the specific subject, and correction means for correcting the position information of the specific subject using the three-dimensional position information of the specific subject. Features.

  ADVANTAGE OF THE INVENTION According to this invention, when designating the position of the viewpoint which concerns on a virtual viewpoint image, the position of the to-be-photographed object can be specified in real space.

The block diagram which shows the structure of an image processing apparatus and a camera apparatus. The schematic diagram which shows arrangement | positioning of a camera apparatus. The figure which shows GUI which sets a virtual camera path | pass. The figure which shows an example of the functional block of an image processing apparatus. The flowchart which shows the process of virtual viewpoint image generation. The flowchart which shows the to-be-photographed object's two-dimensional position calculation process. The figure which showed the bird's-eye view and the example of silhouette extraction. The flowchart which shows a CG scene production | generation process. The figure which showed the example of CG scene production | generation. The flowchart which shows the three-dimensional position calculation process of a specific subject. The figure which showed the specific subject 3D position acquisition example. The flowchart which shows the three-dimensional position acquisition process of an elliptical object.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all the combinations of features described in the present embodiment are not necessarily essential to the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

  In the embodiment described below, a virtual viewpoint is obtained by specifying the position of a subject on the basis of a photographed image taken by a real camera and showing the position of the identified subject on a CG (computer graphic) imitating real space. A mode for facilitating designation of the position of the viewpoint relating to the image will be described.

  Further, as a method of acquiring subject position information in real space, a method of acquiring subject position information by triangulation using a plurality of cameras is generally used. However, such a method has a problem that the processing time becomes long. In addition, there is a problem that it is necessary to use dedicated hardware in order to shorten the processing time. In addition, the height of a subject such as a person to be photographed or a ball changes every moment, and the conventional technology has a problem that the accuracy of the three-dimensional position is low. In the embodiment described below, the subject position can be acquired with high accuracy and in a short time.

  In the embodiment described below, it is assumed that a virtual camera that captures a virtual viewpoint image (hereinafter referred to as a virtual camera) is used, and that the viewpoint position and the direction of the line of sight associated with the virtual viewpoint image are designated. It is also called to specify the position and orientation. Setting various parameters such as the position of the virtual camera, the gazing point, and the angle of view along the time axis is also referred to as setting a virtual camera path.

  In the embodiment described below, the term “image” will be described as including a still image and a moving image unless otherwise specified. That is, in the embodiment described below, the virtual viewpoint image may be a still image or a moving image.

<< Embodiment 1 >>
In the present embodiment, a description will be given using soccer competition as a subject when acquiring position information (also referred to as object position information) of a subject that is a specific object. That is, the description will be made assuming that the shooting target for generating the virtual viewpoint image is a field where soccer is played. In soccer, human subjects such as players are mostly in contact with the field. For this reason, in this embodiment, two-dimensional position information conforming to the field plane is acquired as subject position information, and the processing speed is increased. Also, there are a limited number of subjects that make a three-dimensional movement in soccer. Specifically, a subject that moves three-dimensionally is a ball with a long flight time. Therefore, in the present embodiment, the target for acquiring the three-dimensional position information as the subject information is narrowed down to the ball. As a result, the processing time can be improved and the accuracy of the subject position information in the real space can be increased.

<System configuration>
FIG. 1 is a block diagram showing a system configuration including an image processing apparatus 100 and a camera apparatus 110 in the present embodiment. The image processing apparatus 100 includes a CPU 101, a main memory 102, a storage unit 103, an input unit 104, a display unit 105, an external I / F unit 106, and a bus 107. A CPU (Central Processing Unit) 101 performs arithmetic processing and execution of various programs. The main memory 102 provides the CPU 101 with programs, data, work areas, and the like necessary for processing. The storage unit 103 is a device that accumulates various types of data necessary for image processing programs and GUI (Graphical User Interface) display. For example, a nonvolatile memory such as a hard disk or a silicon disk is used. The input unit 104 is a device such as a keyboard, a mouse, an electronic pen, or a touch panel, and receives an operation input from a user. The display unit 105 displays a screen such as a GUI. An external I / F (InterFace) unit 106 is connected to the camera device 110 and transmits and receives image data and control signal data via the LAN 120. A bus 107 connects the above-described units and performs data transfer.

  The camera device 110 includes a camera group 111 including a plurality of cameras. The camera group 111 is connected to the image processing apparatus 100 via a LAN (Local Area Network) 120. The camera group 111 starts and stops shooting, changes camera settings (such as shutter speed, focal length, and aperture value), and transfers shooting data based on control signals from the image processing apparatus 100.

  In addition to the above, there may be various components of the system configuration, but since it is not the main point of the present embodiment, the description thereof is omitted.

  FIG. 2 is a diagram showing the arrangement of the camera group 111. Here, a case where the camera group 111 is installed in a stadium where sports are performed will be described. A subject 202 exists on the field 201 where the competition is performed. FIG. 2 shows only one subject 202 as an example. The camera group 111 is arranged so as to surround the field 201. In the individual cameras 203 constituting the camera group 111, the focal length, the angle of view (θ °), and the shooting direction are set so that a partial area of the field is accommodated. In addition, a tracking camera 204 is provided separately, and the focal length and the angle of view (φ °) are set so that the entire field can be photographed. Here, θ <φ. In the present embodiment, processing for acquiring position information of the subject 202 is performed based mainly on an image captured by the tracking camera 204 that captures the entire area of the field. That is, processing for acquiring position information of the subject 202 is performed based on an image photographed by one camera. Note that a plurality of tracking cameras 204 may be provided.

<GUI screen>
FIG. 3 is a diagram showing a GUI screen for setting a virtual camera path when generating a virtual viewpoint image. As described above, in order to generate a virtual viewpoint image, it is necessary to set a virtual camera path. The virtual camera path is a path of the virtual camera in which the position, gazing point, and angle of view of the virtual camera are set along the time axis. The virtual viewpoint image is an image in which an image as if taken with this virtual camera is reproduced. In order to set a desired virtual camera path, the user needs to be able to confirm the position of the subject on the field. In the present embodiment, the image processing apparatus 100 performs a process of acquiring subject position information. The image processing apparatus 100 acquires the position information of the subject based on an image captured by one tracking camera 204. And the CG scene which reproduced real space with the CG model is generated using the acquired position information. The user looks at the CG image reproduced using the CG scene and designates the virtual camera path. Then, the image processing apparatus 100 sets a designated virtual camera path, and generates a virtual viewpoint image using a CG model based on the virtual camera path. According to such a process, the image processing apparatus 100 can acquire the position information of the subject for setting the virtual camera path based on the image from one tracking camera 204, so that the cost can be reduced. The position information of the subject can be acquired. Also, the smaller the number of cameras, the smaller the amount of data, so the processing time can be shortened. Although details will be described later, the image processing apparatus 100 acquires three-dimensional position information for a specific subject (for example, a ball) out of the position information of the subject, and is substantially two-dimensional for other subjects. Get location information. According to such a process, highly accurate position information can be acquired in a short time.

  The GUI screen 300 shown in FIG. 3A includes a CG window 310, operation buttons 320, and a camera path setting dialog 330. The CG window 310 is a window for visualizing various three-dimensional shape CG models (hereinafter referred to as CG models) including a field CG model 311, a player CG model 312, and a ball CG model 313. Each CG model is composed of polygonal surfaces and is visualized using an existing CG rendering method such as a ray tracing method or a scan line method. In the present embodiment, a CG scene that reproduces the field on which the camera group 111 is to be captured in a desired time zone is displayed in the CG window 310. The user locates and moves the virtual camera while checking the position of each CG model indicating a subject such as a player visualized in the CG window 310, and designates the virtual camera path of the virtual viewpoint image. In the present embodiment, a process of acquiring the position information of the subject (CG model) visualized in the CG window 310 in a short time with high accuracy will be described. In particular, in the present embodiment, the subject position information of the player CG model 312 and the ball CG model 313 is acquired with high accuracy in a short time. Details will be described later.

  The operation buttons 320 are used for reading necessary input data and starting player tracking. Player tracking is to track the position of each player as a subject over time. Specifically, tracking data indicating the subject position information of each player over time is collected. The operation buttons 320 include a CG model reading button 321, a tracking data application button 322, a virtual viewpoint image generation button 323, and a player tracking button 324.

  The camera path setting dialog 330 is used when setting the virtual camera path in detail. The camera path setting dialog 330 includes a virtual camera path setting button 331 and a ball height setting button 332.

  The GUI screen 350 in FIG. 3B is a screen in which the CG window 310 is changed to the tracking window 360 when the player tracking button 324 of the operation buttons 320 in FIG. In the tracking window 360, an image taken by the tracking camera 204 is displayed, or a bird's eye view image of the field viewed from above is displayed. Details will be described later. Based on the bird's-eye view image displayed in the tracking window 360, the user performs settings necessary for tracking the subject position information of the player and the ball. In the GUI screen 350 in FIG. 3B, the camera path setting dialog 330 in FIG. 3A transitions to the tracking dialog 370 in conjunction with the transition to the tracking window 360. The details of FIG. 3 will be described as appropriate in connection with the processing described later.

<Functional block diagram>
FIG. 4 is a diagram illustrating an example of a functional block diagram of the image processing apparatus 100 according to the present embodiment. The image processing apparatus includes an image data storage unit 401, an image time acquisition unit 402, an image data extraction unit 403, an image conversion unit 404, a two-dimensional position information acquisition unit 405, a CG model storage unit 406, and a CG scene generation unit 407. In addition, it includes a three-dimensional position information acquisition unit 408, a correction unit 409, a display control unit 410, a virtual camera path setting unit 411, and a virtual viewpoint image generation unit 412.

  The storage unit 103 illustrated in FIG. 1 functions as an image data storage unit 401 and a CG model storage unit 406. Note that an external device connected to a network (not shown) may be virtually used to function as the image data storage unit 401 and the CG model storage unit 406. The CPU 101 functions as each unit illustrated in FIG. 4 when the CPU 101 performs processing according to the software program stored in the main memory 102 or the storage unit 103. Thus, in the present embodiment, the configuration of the image processing apparatus 100 will be described by processing by software. However, the processing may be realized by hardware dedicated to performing similar processing, or by a combination of such hardware and software. 4 shows a form included in the image processing apparatus 100, but a form in which processing of each part shown in FIG. 4 is distributed and processed by a plurality of apparatuses may be adopted. .

  The image data storage unit 401 stores image data captured by the tracking camera 204. The image time acquisition unit 402 acquires an image time (including a start time and an end time) designated by the user. The image data extraction unit 403 extracts image data corresponding to the image time acquired by the image time acquisition unit 402 from the image data storage unit 401. The image conversion unit 404 converts the image indicated by the image data extracted by the image data extraction unit 403 (image captured by the tracking camera 204) into a bird's eye view image.

  A two-dimensional position information acquisition unit (first acquisition unit) 405 acquires subject position information of each subject using the bird's eye view image converted by the image conversion unit 404. Here, the height (the vertical coordinate of the field) is fixed at a predetermined value (for example, “0”), and substantially two-dimensional position information is acquired. That is, assuming that each subject is located on the field plane, two-dimensional subject position information of each subject is acquired. Here, in order to facilitate understanding, it is described as two-dimensional subject position information, but the coordinates indicating the height are substantially fixed to “0” as described above. Dimensional subject position information is acquired. The two-dimensional position information acquisition unit 405 acquires tracking data indicating subject position information over time.

  The CG model storage unit 406 stores various CG models. For example, a field CG model 311, a player CG model 312, and a ball CG model 313 are stored. The CG scene generation unit 407 generates a CG scene using the CG model stored in the CG model storage unit 406 based on the tracking data (subject position information with time) acquired by the two-dimensional position information acquisition unit 405. To do.

  A CG scene is a material for producing a CG image. A CG image is reproduced by placing a three-dimensional shape model of a subject or background in a virtual space, then setting a virtual camera and rendering an image taken from the virtual camera. Generating a CG scene is equivalent to arranging a three-dimensional shape model of a subject or background in the virtual space. The CG scene includes the concept of time (animation).

  A three-dimensional position information acquisition unit (second acquisition unit) 408 acquires three-dimensional position information of a specific subject using the CG scene generated by the CG scene generation unit 407. An example of the specific subject is a ball that may move over the field. When the ball is in the air, there is a high possibility that the position information acquired by the two-dimensional position information acquisition unit 405 is not correct. Therefore, the three-dimensional position information acquisition unit 408 acquires the coordinate value of the position information (three-dimensional position) of the specific subject. Details will be described later.

  The correction unit 409 corrects the CG scene generated by the CG scene generation unit 407. For example, the position information of a specific subject in the CG scene is changed so that the three-dimensional position information acquisition unit 408 can acquire the three-dimensional position information. Details will be described later. The correction unit 409 corrects the tracking data of the specific subject (ball) using the coordinate value of the subject position information (three-dimensional position information) acquired by the three-dimensional position information acquisition unit 408. Then, the CG scene generated by the CG scene generation unit 407 is corrected using the corrected tracking data. The display control unit 410 controls the display unit 105 to display the CG scene generated by the CG scene generation unit 407, the CG scene corrected by the correction unit 409, or a CG image using the CG scene.

  The virtual camera path setting unit 411 functions as a reception unit that receives designation from the user. Then, a virtual camera path designated by the user is set. The user confirms the position of each subject from the displayed CG scene and designates a desired virtual camera path. The virtual viewpoint image generation unit 412 generates a CG virtual viewpoint image using a CG scene according to the virtual camera path set by the virtual camera path setting unit 411. The generated virtual viewpoint image is displayed on the display unit 105, and the user can confirm whether or not the virtual camera path designated by the user is desired.

  Note that if the virtual camera path specified by the user is desired, a virtual viewpoint image of an actual image according to the specified virtual camera path using images taken by the cameras 203 constituting the camera group 111. Is generated.

<Flowchart>
FIG. 5 is a flowchart showing the process until the above-described units generate a virtual viewpoint image using a CG scene.

  In step S <b> 501, the CPU 101 changes the camera settings so that exposure at the time of shooting is appropriate and transmits a shooting start signal to the camera device 110. The camera device 110 starts shooting in response to the reception of the shooting signal, and transfers the shot image data to the image processing device 100 via the LAN 120. The camera device 110 also transfers the image data of the tracking camera 204. The transmitted image data is stored in the image data storage unit 401.

  In step S502, the image time acquisition unit 402 acquires the image time set by the user. The image time is a time at which an image that is a target of virtual viewpoint image generation is taken. The image time is set by the user through a UI screen (not shown). In step S <b> 503, the image data extraction unit 403 extracts image data corresponding to the image time from the image data storage unit 401. In step S <b> 503, the CPU 101 activates the GUI screen 300 in FIG. 3A and displays it on the display unit 105. Immediately after startup, nothing is displayed in the CG window 310.

  In step S505, the two-dimensional position information acquisition unit 405 acquires the two-dimensional position information of the subject. For example, the player tracking button 324 on the GUI screen 300 displayed in step S504 is pressed by the user. In response to pressing of the player tracking button 324, the CPU 101 causes the screen displayed on the display unit 105 to transition to the GUI screen 350 shown in FIG. That is, the CPU 101 displays the GUI screen 350 including the tracking window 360 and the tracking dialog 370 on the display unit 105. The two-dimensional position information acquisition unit 405 acquires the image data extracted by the image data extraction unit 403, that is, all frames (image data) included in the image time acquired in step S502. Then, in those frames (image data), the subject position information (two-dimensional position) of the subject 202 along the field plane is acquired based on the image of the tracking camera 204. In this step, not the three-dimensional position information but the two-dimensional position information is acquired as the subject position information, so that the processing can be executed at high speed. Details of this processing will be described later. The acquired subject position information (two-dimensional position) is stored in the storage unit 103.

  In step S506, the CG scene generation unit 407 generates a three-dimensional CG scene using the acquired subject position information (two-dimensional position) and the existing CG model stored in the CG model storage unit 406. . Details of this process will also be described later.

  In step S507, the three-dimensional position information acquisition unit 408 extracts a specific subject (ball) from the subjects, and calculates the three-dimensional position of the specific subject. The ball may be located above the field and not the ground during competition. Therefore, the three-dimensional position information acquisition unit 408 acquires such three-dimensional subject position information of the specific subject. Thereby, the position acquisition accuracy of the ball is improved. In addition, since the processing target for obtaining the three-dimensional position is narrowed down to the ball, the processing can be executed at high speed. Details of this process will also be described later.

  In step S508, the virtual camera path setting unit 411 performs virtual camera path setting. For example, the user designates a virtual camera path in the CG scene displayed in the CG window 310 which is a display area for displaying the CG in FIG. For example, a configuration may be adopted in which a user's drag operation or touch operation on the CG window 310 is received and a trajectory corresponding to these operations is designated as a virtual camera path. Then, in response to pressing of the virtual camera path setting button 331, the virtual camera path setting unit 411 sets the designated virtual camera path. The virtual camera path setting is performed in a three-dimensional CG scene. As the method, for example, a method of sampling a virtual camera position at several points in the time axis direction and interpolating between the positions using a spline function or the like can be used.

  In step S509, the virtual viewpoint image generation unit 412 generates a virtual viewpoint image according to the set virtual camera path. Specifically, when the virtual viewpoint image generation button 323 in FIG. 3A is pressed by the user, the virtual viewpoint image generation unit 412 generates a virtual viewpoint image based on the set virtual camera path. In the virtual viewpoint image generation, an image viewed from a virtual camera can be generated for a three-dimensional shape of a subject using a known computer graphics technique.

  Whether or not the user can reproduce the desired virtual viewpoint image by viewing the virtual viewpoint image of the CG scene when the virtual camera path for generating the virtual viewpoint image of the actual image is designated by the processing described above. Can be easily determined.

<Subject position information (two-dimensional position) acquisition processing>
Next, details of the subject position information (two-dimensional position) acquisition process in step S504 will be described with reference to the flowchart of FIG. FIG. 6 is a flowchart showing the process of S504.

  In step S <b> 601, the image conversion unit 404 acquires the image of the tracking camera 204 extracted by the image data extraction unit 403. For example, image acquisition is executed when the player tracking button 324 in FIG. 3A is pressed by the user. The acquired image is displayed in the tracking window 360 of FIG. For example, the tracking window 360 displays the tracking camera image 701 at the specified image time, which is taken by the tracking camera 204, as in the tracking camera image 701 in FIG.

  In step S602, the image conversion unit 404 acquires position information of the four corners of the field. The four corners correspond to the positions of the four corners of the bird's-eye view image when the tracking camera image is converted into a bird's-eye view image viewed from above the field. The position information of the four corners is acquired, for example, when the user who has seen the tracking camera image 701 displayed in the tracking window 360 specifies the four corners of the field. The corner is a position indicated by a corner 705 in FIG. 7A, and the user may specify with a mouse or a touch operation. Of course, the position of the corner may be detected using a straight line extraction method such as Hough transform.

  In step S603, the image conversion unit 404 performs projective conversion on the tracking camera image, and generates a bird's-eye view image of the field viewed from above. The size of the soccer field is prescribed. For this reason, the coordinate values of the four corners can be directly converted into physical position coordinates (world coordinates). The focal length of the tracking camera 204 is known. When the focal length is known, the physical position (world coordinate) of the tracking camera 204 can be calculated from the distortion angle of the quadrilateral that forms the field.

Hereinafter, the image of the tracking camera 204 is represented by “Image (X, Y)”. The subject shown in the tracking camera image is located on the two-dimensional coordinates (screen coordinates) on the tracking image, but physically located on the three-dimensional coordinates (world coordinates). . Therefore, the tracking image Image (X, Y) (two-dimensional coordinates) can be said to be an image according to Equation 1. X and Y indicate pixel positions (two-dimensional coordinates) of the tracking image.
Image (X, Y) = Screen (Proj * View * Pos (x, y, z)); Equation 1
here,
Pos (x, y, z): three-dimensional position of the subject (three-dimensional coordinates)
View: Viewpoint transformation matrix (determines camera position and direction)
Proj: Two-dimensional projection matrix (determines the angle of view of the camera)
Screen (): Screen coordinate conversion function.

The bird's eye view image is represented by “ImageV (X, Y)”. The subject shown in the bird's-eye view image is also located on the two-dimensional coordinates (screen coordinates) on the tracking image on the bird's-eye view image, but physically located on the three-dimensional coordinates (world coordinates). It will be. Therefore, it can be said that the bird's-eye view image ImageV (X, Y) (two-dimensional coordinates) is an image according to Equation 2. X and Y indicate pixel positions (two-dimensional coordinates) of the bird's eye view image.
ImageV (X, Y) = Screen (Proj * ViewV * Pos (x, y, z)); Equation 2
here,
Pos (x, y, z): Three-dimensional position of the subject ViewV: View point transformation matrix (determines the position and direction of the camera)
Proj: Two-dimensional projection matrix (determines the angle of view of the camera)
Screen (): Screen coordinate conversion function.

  When Pos (x, y, z) is made to correspond using Equation 1 and Equation 2, projective transformation from Image (X, Y) to ImageV (X, Y) can be performed. That is, the pixel Image (X, Y) (screen coordinates) at a certain position on the tracking camera image can be converted into the first point Pos (x, y, z) in the world coordinates using Equation 1. . Further, by converting the first point Pos (x, y, z) of the world coordinate system using Expression 2, the tracking image Image (X, Y) is converted into the bird's eye view image ImageV (X, Y) ( Screen coordinates). In the present embodiment, the z coordinate (indicating the height) at the position Pos (x, y, z) of the world coordinate is fixed on the field plane (fixed at 0). That is, in the present embodiment, the z coordinate (height) at which the subject (for example, a player) is located is handled as a fixed value fixed at a predetermined height (field surface). By performing processing in this way, it is possible to generate a bird's eye view image using an image of one tracking camera.

  As described above, there are methods such as triangulation using a plurality of cameras in order to obtain the z-coordinate of the subject, but processing time is required due to an increase in the amount of data. In the present embodiment, the subject is treated as being fixed on the field surface, and the image of one camera is converted into a bird's eye view image. By generating the bird's-eye view image, it is possible to acquire subject position information (two-dimensional position) on the field plane of each subject.

  The image conversion unit 404 performs the above-described image conversion process, and converts the tracking camera image 701 in FIG. 7A to the bird's eye view image 702 in FIG. 7B. The converted bird's-eye view image 702 is displayed in the tracking window 360 of FIG.

  In step S604, the two-dimensional position information acquisition unit 405 extracts an intermediate value in units of pixels in all frames of the bird's eye view image 702 generated in step S603, and generates a background image in which each pixel is expressed by the extracted intermediate value. To do. This background image is used to generate a subject silhouette image to be described later. The background image corresponds to an image in which no subject is shown.

  In step S605, the two-dimensional position information acquisition unit 405 generates a subject silhouette image. Specifically, the two-dimensional position information acquisition unit 405 performs a pixel-by-pixel difference between the bird's eye view image 702 and the background image generated in step S604 in all frames of the bird's eye view image 702 generated in step S603. Then, a subject silhouette image in which each pixel is expressed by the difference pixel value is generated. FIGS. 7C and 7D show silhouette images 703 and 704 having different frames (time), respectively. It can be seen that the white areas of the silhouette images 703 and 704 are areas corresponding to the silhouette of the subject (player or ball), and only the subject is extracted.

  In step S606, the two-dimensional position information acquisition unit 405 labels the subject area of the silhouette image of each frame. For example, the two-dimensional position information acquisition unit 405 gives a unique label number to an isolated subject silhouette area. FIG. 7C and FIG. 7D show that labels M1 to M4 are assigned to the subject silhouette. In step S606, the two-dimensional position information acquisition unit 405 assigns labels to the subject area in order from the upper left to the lower right of the screen for each frame.

  In step S607, the two-dimensional position information acquisition unit 405 adjusts the subject and the label number set in units of frames so as to maintain consistency in all frames. In the previous step S606, the label number is simply assigned according to the position in the frame without considering the relationship with other frames. In step S607, adjustment is performed so that the same label number is assigned to the same subject throughout all frames in consideration of the relationship with other frames. For example, if the label number assigned to a certain subject X is shifted in frames t0 and t1, the label number assigned to each frame is changed so that the label number assigned to the subject X becomes the same label number. To do. Specifically, when the position of the subject area of the label MX1 in the frame t0 is close to the position of the subject area of the label MX2 in the frame t1, a process of rewriting the label MX2 of the frame t1 with the label MX1 is performed. The labeling adjustment may be performed using a bird's eye view image in addition to the silhouette image. For example, the labeling may be adjusted by recognizing and using the face, back number, etc. of the subject. Through the above processing, the position of a specific subject X in all the frames of the bird's eye view image 702 can be specified.

  In step S608, the two-dimensional position information acquisition unit 405 adds an attribute to the subject. The attribute means additional information such as a player's team, keeper, referee, and ball. An example of adding an attribute to a subject based on user designation will be described with reference to FIG. In the tracking window 360 of the GUI screen 350 in FIG. 3B, subjects M1, M2, and M3 to which label numbers are assigned are displayed. The image displayed in the tracking window 360 is an image in which the subject silhouette and the label number are superimposed on the bird's eye view image. Here, for example, when the user clicks or touches the subject M1 with the mouse, a combo box 371 for designating the attribute of the subject M1 is displayed in the tracking dialog 370. The user selects an attribute from the combo box 371. In this embodiment, a soccer scene is assumed. As an attribute, any one of a team A field player, a team B field player, a team A goalkeeper, a team B goalkeeper, a referee, and a ball is used. Will be granted. When an attribute is selected, the CG scene generation unit 407 acquires a CG model according to the attribute from the CG model storage unit 406 and previews it. The CG model will be described later. The two-dimensional position information acquisition unit 405 gives the selected attribute to the subject M1. This series of processing is performed on all subjects. Note that, here, an example in which attribute assignment is performed based on user designation has been described as an example, but attribute assignment can also be automatically processed using image recognition using machine learning. Further, by using the feature that the ball is circular or elliptical, the attribute of the ball can be automatically given.

  In step S609, the two-dimensional position information acquisition unit 405 calculates the two-dimensional position of the subject for each label number. Specifically, the XY coordinate of the minimum value in the predetermined direction (Y coordinate) of each subject silhouette is set as the subject position. The XY coordinate of the minimum value in the predetermined direction corresponds to the XY coordinate of the foot of each subject (minimum value of the Y coordinate). By using the XY coordinates of the foot of the subject silhouette as the subject position, it is possible to acquire subject position information that is not easily influenced by the posture of the subject. The XY coordinates are defined along the axes shown in the silhouette images 703 and 704 in FIGS. 7C and 7D. In this way, subject position information (two-dimensional position) through all target frames is acquired for each subject. The subject position information through all the target frames is also referred to as tracking data indicating temporal position information. Thereafter, the user presses the tracking end button 372 of the tracking dialog 370 on the GUI screen 350 in FIG. Then, the two-dimensional position information acquisition unit 405 stores tracking data in which information on the two-dimensional position of the subject with time is recorded in the storage unit 103. The CPU 101 also transitions the display screen to the GUI screen 300 shown in FIG. 3A, and ends a series of processing. At this time, the CG window 310 remains blank. In the processing of FIG. 6, the processing is uniformly performed on all subjects, but the target for generating tracking data may be limited using attributes. For example, the processing can be performed more efficiently by setting only the subject located around the subject of the ball attribute, which is important in the virtual viewpoint image, as the tracking target.

  The above is the details of the process of acquiring the two-dimensional position of the subject in step S505 in FIG. Next, details of the CG scene generation processing in step S506 in FIG. 5 will be described with reference to FIG.

<CG scene generation processing>
FIG. 8 is a flowchart showing details of the CG scene generation processing in step S506 of FIG. In step S801, the CG scene generation unit 407 detects that the user has pressed the CG model reading button 321 in FIG. 3A, and reads the CG model of the field and the subject, respectively.

  FIG. 9 is a diagram illustrating an example of a CG model. The field model 907 is made up of polygons, and field lines, which are shooting target areas of the camera group 111, are represented as textures. As shown in FIG. 9, the subject model includes a team A model 901, a team B model 902, a keeper model (team A) 903, a keeper model (team B) 904, a referee model 905, and a ball model 906. All models are composed of polygons, and uniforms and ball patterns are expressed as textures. As many models as the types of uniforms are prepared so that the user is not confused when setting the virtual camera path. Of course, it is desirable to have more types, such as preparing for each number. For example, in the case of soccer, the team A model 901 and the team B model 902 may prepare 10 field players for each team.

  In step S <b> 802, the CG scene generation unit 407 detects that the user has pressed the tracking data application button 322 in FIG. 3A, and reads tracking data in which the two-dimensional position of the subject is recorded.

  In step S803, the CG scene generation unit 407 refers to the read tracking data and duplicates a model according to the attribute. For example, the CG scene generation unit 407 counts the number of subjects nA having the team A attribute. Then, nA team A models 901 are duplicated. Similarly, the number of subjects nB having the attribute of team B is counted, and nB team B models 902 are duplicated. In addition, the number nS of subjects having a referee attribute is counted, and nS reference models 905 are duplicated. Here, the CG models of each team and the referee are shown as the same model. However, in the case where a CG model is prepared for each subject, it is necessary to duplicate each subject model by the corresponding number. Good. Similarly, the keeper model is duplicated.

  In step S804, the CG scene generation unit 407 sets two-dimensional position information for all CG models except the field model 907 based on the tracking data read in step S802. In FIG. 9, it can be confirmed that each model is duplicated as necessary and arranged on the field model 907. In addition, since the tracking data is two-dimensional position information recorded in units of frames, a CG scene in which the subject moves as the frame advances can be constructed.

  In step S805, the display control unit 410 displays the generated CG scene on the CG window 310 of FIG. With the above processing, the CG scene generation processing ends. In other words, the display control unit 410 is CG which is information indicating an object such as a player who is a subject at a position corresponding to the acquired position information on the field model 907 which is an image indicating a shooting target area of the camera group 111. Display the model superimposed.

  At this time, the height information (Z coordinate value) of all subjects is set to zero. For this reason, the subject position information may not be correct for a ball that may not be in contact with the field. For this reason, the position of the ball may be incorrect in the generated CG scene. Accordingly, in step S507 in FIG. 5, the process of acquiring subject position information (three-dimensional information) of a specific subject (ball) is performed.

<Acquisition processing of subject position information (three-dimensional information) of a specific subject>
FIG. 10 is a flowchart showing details of the three-dimensional position acquisition process of the ball in step S507 of FIG. This process corresponds to the process of correcting the position information of a specific subject (ball) that may have been set incorrectly in the process of FIG. A virtual camera pass may be set for a scene where the ball does not exist in the air. For example, when it is desired to reproduce a dribble scene. In this way, it may be assumed that acquisition of the three-dimensional position of the ball is clearly unnecessary. Therefore, in the present embodiment, a process in which the user acquires the three-dimensional position information of a specific subject by pressing the ball height setting button 332 of the camera path setting dialog 330 in FIG. Start. Note that the process of FIG. 10 may be started without relying on the user instruction.

  In step S1001, the three-dimensional position information acquisition unit 408 acquires the silhouette area of the ball in the processing target frame in the bird's eye view image. In step S1002, the three-dimensional position information acquisition unit 408 calculates the area (number of pixels) S1 of the silhouette region of the ball. In step S1003, the three-dimensional position information acquisition unit 408 calculates a straight line L (see FIG. 11A) that connects the position of the tracking camera 204 in the three-dimensional space and the center coordinates of the silhouette of the ball. FIG. 11 is a diagram for explaining the outline of the processing for acquiring the three-dimensional position information. As shown in FIG. 11A, the correct position of the ball exists anywhere on the straight line L. Note that the position and shooting parameters of the tracking camera 204 in the three-dimensional space are known, and the straight line L is calculated using the known information and the center coordinates of the silhouette.

  Here, the distance between the tracking camera 204 and the ball and the area of the ball silhouette region are in an inversely proportional relationship. FIG. 11A and FIG. 11B show an example in which the ball position on the straight line L is varied. Comparing the distance between the tracking camera 204 and the ball in FIGS. 11 (a) and 11 (b), FIG. 11 (b) is longer. Therefore, it can be seen that the silhouette area of the ball in the bird's-eye view image is smaller in FIG. 11B than in FIG. In the present embodiment, the three-dimensional position information of the ball is acquired using the size of the silhouette area of the ball.

  In step S1004, the three-dimensional position information acquisition unit 408 arranges the CG tracking camera 1101 at the same three-dimensional position as the tracking camera 204 in the CG scene generated in step S506. At this time, the shooting parameters (such as focal length) of the tracking camera 204 and the CG tracking camera 1101 are made common.

  In step S1005, the three-dimensional position information acquisition unit 408 sets a straight line L in the CG scene. Then, the ball model 906 is arranged on the straight line L. That is, processing for reproducing an image having the same composition as the image captured by the tracking camera 204 using a CG scene is performed. Then, a position where the area of the ball silhouette in the reproduced CG scene becomes equal to the ball silhouette of the bird's eye view image obtained by converting the real image is estimated as the three-dimensional position of the ball. In the present embodiment, processing for obtaining an appropriate ball position is performed while shifting the arrangement position of the ball model 906. In step S1005, the ball model 906 is disposed at a position close to the CG tracking camera 1101.

  In step S1006, the correction unit 409 renders the CG image viewed from the CG tracking camera 1101. In step S1007, the correction unit 409 performs projective transformation on the rendered CG image and generates a CG bird's-eye view image. The generation of the CG bird's-eye view image can be performed by the same method as the above-described generation of the bird's-eye view image.

  In step S1008, the three-dimensional position information acquisition unit 408 calculates a silhouette area (number of pixels) S2 of the ball model in the CG bird's eye view image. The generation of the silhouette image in the CG bird's-eye view image can also be performed by the same method as the method for generating the silhouette image from the bird's-eye view image described above. In step S1009, the three-dimensional position information acquisition unit 408 calculates a difference between the area S1 and the area S2, and determines whether the difference value is less than a predetermined threshold value. If the difference value is less than the predetermined threshold value, it is considered that the silhouettes match, and the series of processes for the processing target frame is terminated, and the process proceeds to step S1011. If the difference value is greater than or equal to the threshold value, the process proceeds to step S1010. In step S1010, the correction unit 409 updates the position of the ball model 906, and proceeds to step S1006.

  An outline of steps S1005 to S1010 will be described with reference to FIG. By arranging the ball model on the straight line L in the CG scene and generating the CG bird's eye view image, the silhouette of the ball model 906 in the CG scene and the silhouette of the ball in the real space can be compared. When both sizes match, the three-dimensional position of the ball model 906 at that time becomes the three-dimensional position of the ball in real space. In step S1010, the position of the ball model 906 on the straight line L is adjusted so that the distance between the three-dimensional coordinate value of the ball model 906 and the CG tracking camera is increased.

  In step S1011, it is determined whether processing of all frames has been completed. If the processing of all frames has not been completed, the process proceeds to step S1012, the unprocessed frame is set as the processing target frame, the process returns to step S1001, and the process is repeated. Through the above processing, an accurate three-dimensional position of the ball can be set.

  Thereafter, as described above, a CG scene in which subject position information of a specific subject (ball) is corrected is generated. In addition, a virtual camera path is set based on the generated CG scene.

  As described above, in the present embodiment, the subject position information is acquired with high accuracy and in a short time from one camera image by narrowing down the target for acquiring the three-dimensional subject position information to a specific subject. It becomes possible. Further, since the position of the subject that is the player is indicated by the CG in association with the field, the user can specify the position of the subject in the real space. Therefore, the user can easily set a virtual camera path for obtaining a desired virtual viewpoint image.

<< Embodiment 2 >>
In the present embodiment, the acquisition of the subject position will be described using a rugby game with an elliptical ball as a theme. Since most of the processing is the same as that of the first embodiment, the description of the overlapping parts is omitted.

  FIG. 12 is a flowchart showing details of the three-dimensional position information acquisition process in step S506 of FIG. Except for steps S1202, S1208, and S1209, the processing is the same as that shown in FIG. In step S1202, the three-dimensional position information acquisition unit 408 calculates the area (number of pixels) S1 and the eccentricity r1 of the silhouette region of the ball. The eccentricity represents the ratio of the major axis a and the minor axis b of the ellipse, and is a constant value if the ball rotation direction is the same. When the ball is elliptical, even if the ball is in the same three-dimensional position, the area of the silhouette region differs depending on the rotation direction of the ball. Therefore, in the present embodiment, processing is performed in consideration of the eccentricity. The eccentricity can be calculated by the following formula 3.

  In step S1208, the three-dimensional position information acquisition unit 408 calculates the silhouette area (number of pixels) S2 and the eccentricity r2 of the ball model in the CG bird's-eye view image. In step S1209, the correction unit 409 performs only rotation without changing the position of the ball model, and repeats rotation until r1 and r2 are equal. Thereby, the rotation direction of the ball model and the ball in the real space coincide. Then, the silhouette areas are compared in step S1210 with the rotation directions being matched. Thus, according to the processing of this embodiment, an accurate three-dimensional position can be set even for an elliptical ball such as rugby. In this example, the method for determining the rotation direction based on the eccentricity is described, but any parameter representing an elliptical shape may be used.

  As described above, by considering the rotation direction, subject position information specialized for virtual viewpoint image generation can be obtained with high accuracy and in a short time from a single camera image even in a game using an elliptical ball. can do.

<< Other Embodiments >>
In the embodiment described above, the soccer competition and the rugby competition have been described as the theme, but the present invention is not limited to these. It is possible to apply the present invention to a form in which an arbitrary ball game is performed on the field. Further, the present invention can be applied not only to ball games but also to various shooting objects such as skating, sumo, and concerts. Further, the subject from which the three-dimensional position information is acquired has been described by taking the shape of a circular (spherical) or elliptical subject as an example, but is not limited thereto. For example, a subject having a specific shape, such as a badminton shuttle, can be used as a subject to acquire three-dimensional position information.

  Further, in the CG window 310, a back number or a face image for identifying an individual may be mapped to the player CG model 312 as a texture. The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Tracking camera 204
Image conversion unit 404
Two-dimensional position information acquisition unit 405
3D position information acquisition unit 408

Claims (22)

An image processing apparatus that accepts designation of a viewpoint position related to a virtual viewpoint image generated using a plurality of images photographed by a plurality of cameras,
An acquisition means for acquiring position information of an object shown in the image based on an image obtained by imaging the imaging target areas of the plurality of cameras;
Display control means for displaying information indicating the object at a position corresponding to the position information acquired by the acquisition means in the computer graphic indicating the imaging target area;
Accepting means for accepting designation of the position of the viewpoint related to the virtual viewpoint image in a state where the information indicating the object is displayed by the display control means;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the information displayed by the display control unit is a computer graphic.   The image processing apparatus according to claim 1, wherein the shooting target areas of the plurality of cameras are fields where soccer or rugby is played.   The reception unit receives specification of a position of a viewpoint related to a virtual viewpoint image based on an operation on a display area that displays an image indicating the imaging target area and information indicating the object. 4. The image processing device according to any one of items 1 to 3.   5. The image processing apparatus according to claim 1, wherein the acquisition unit acquires the position information of the object based on an image obtained by capturing the imaging target region from the sky.   6. The image processing apparatus according to claim 1, further comprising a conversion unit configured to convert an image obtained by photographing the photographing target region into an image obtained by capturing the photographing target region from above. Conversion means for converting an image of the field into a bird's eye view image of the field viewed from above;
First acquisition means for acquiring two-dimensional position information of a subject from the bird's eye view image;
Generating means for generating a CG scene in which a region of the field including the subject is reproduced by a CG (computer graphic) model using the two-dimensional position information;
Second acquisition means for acquiring three-dimensional position information of a specific subject using the CG scene;
An image processing apparatus comprising: correction means for correcting position information of the specific subject using three-dimensional position information of the specific subject.   The image processing apparatus according to claim 7, wherein the correction unit corrects the CG scene using the corrected position information of the specific subject. Display control means for displaying the CG scene on a screen;
9. The image processing apparatus according to claim 7, further comprising setting means for setting a designated virtual camera path.   The image processing apparatus according to claim 9, further comprising a first image generation unit configured to generate a virtual viewpoint image using the CG scene according to the set virtual camera path. Third acquisition means for acquiring a plurality of images obtained by photographing the field from a plurality of viewpoints;
The image processing apparatus according to claim 9, further comprising a second image generation unit configured to generate a virtual viewpoint image using the plurality of images according to the set virtual camera path.   The image processing apparatus according to claim 7, wherein the bird's eye view image is an image converted from an image obtained by photographing the field with a single camera. The first acquisition means includes
Extract the silhouette of the subject from the bird's eye view image,
The image processing apparatus according to claim 7, wherein coordinates of a minimum value in a predetermined direction of the silhouette are acquired as two-dimensional position information of each subject.   The image processing apparatus according to claim 13, wherein the second acquisition unit acquires the three-dimensional position information using an area of a silhouette of the specific subject.   The image processing apparatus according to claim 14, wherein the second acquisition unit acquires the three-dimensional position information by further using a parameter indicating the shape of the specific subject. The generation unit generates a CG image having the same composition as an image obtained by photographing the field using the CG scene, generates a CG bird's eye view image from the generated CG image,
The second acquisition means includes a position of a CG model where an area of the silhouette of the specific subject in the bird's-eye view image and an area of the silhouette of the CG model of the specific subject in the CG bird's-eye view image are less than a predetermined threshold. The image processing apparatus according to claim 13, wherein the image processing device is acquired as three-dimensional position information of the specific subject.   The image processing apparatus according to claim 7, wherein the specific subject includes a circular or elliptical subject.   The image processing apparatus according to any one of claims 7 to 17, wherein the CG model uses the same number of types as or more than the types of uniforms of the subject.   The image processing apparatus according to claim 7, wherein the correction unit performs the correction in accordance with an instruction from a user. Converting an image of the field into a bird's eye view image of the field viewed from above;
Obtaining two-dimensional position information of the subject from the bird's eye view image;
Using the two-dimensional position information to generate a CG scene in which a field area including the subject is reproduced by a CG (computer graphic) model;
Obtaining three-dimensional position information of a specific subject using the CG scene;
And correcting the position information of the specific subject using three-dimensional position information of the specific subject. An image processing method by an image processing apparatus that accepts designation of a viewpoint position related to a virtual viewpoint image generated using a plurality of images photographed by a plurality of cameras,
An acquisition step of acquiring position information of an object shown in the image based on an image obtained by imaging the imaging target areas of the plurality of cameras;
In the computer graphic indicating the shooting target area, a display control step of displaying information indicating the object at a position corresponding to the position information acquired in the acquisition step;
An accepting step of accepting designation of a position of a viewpoint related to a virtual viewpoint image in a state in which information indicating the object is displayed in the display control step;
An image processing method comprising:   The program for functioning a computer as each means as described in any one of Claims 1-19.

Images