JP7034666B2 - Virtual viewpoint image generator, generation method and program - Google Patents

Description

The present invention relates to image processing for generating a virtual viewpoint image from a plurality of viewpoint images.

There is a virtual viewpoint image technology as a technology for reproducing an image from a camera (virtual camera) that does not actually exist and is virtually arranged in a three-dimensional space by using images taken by a plurality of real cameras. Virtual viewpoint image technology is expected as a more realistic image expression in sports broadcasting and the like. In the generation of the virtual viewpoint image, the image taken by the actual camera is taken into the image processing device, and the shape of the object is first estimated. Next, the user determines the movement path of the virtual camera based on the result of shape estimation, and reproduces the image taken from the virtual camera. Here, for example, if the shooting scene is a soccer match, when determining the movement route of the virtual camera, it is necessary to estimate the shape of the player or the ball, which is an object, in the entire field where the soccer is played. .. However, when the object shape estimation process is performed on the entire area of a wide field, the transfer time and the shape estimation process time of the multi-viewpoint video data taken by the actual camera are increased. In order to realize a more powerful and realistic match broadcast, it is important to broadcast a virtual viewpoint image of a shoot scene as a replay image in a timely manner during the match. The increase in video transfer time and shape estimation processing time is an obstacle to the generation of highly real-time virtual viewpoint images.

In this regard, the video data taken by the actual camera is held at different resolutions, the shape is first estimated with the low resolution video, the result is used as the initial value, and then the shape is estimated with the high resolution video. A technique for shortening the processing time by repeating the process has been proposed (Patent Document 1).

Japanese Unexamined Patent Publication No. 5-126546

However, although the method of Patent Document 1 can shorten the shape estimation processing time, it cannot shorten the transfer time of the multi-viewpoint video data taken by the actual camera to the image processing device.

The virtual viewpoint image generation device according to the present invention is a plurality of captured images obtained by a plurality of first imaging devices that capture fields from different directions, depending on the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint. It is obtained by a first acquisition means for acquiring a plurality of captured images used for generating a first virtual viewpoint image, and one or a plurality of second imaging devices that capture at least a part of the field from different directions. Alternatively, whether or not to generate a second virtual viewpoint image having a higher image quality than the first virtual viewpoint image based on the plurality of captured images is generated based on the plurality of captured images acquired by the first acquisition means . A determination means for determining according to the evaluation result of the first virtual viewpoint image, and a second acquisition means for acquiring one or a plurality of captured images obtained by the second imaging device according to the determination by the determination means. And, based on the plurality of captured images acquired by the first acquisition means and one or a plurality of captured images acquired by the second acquisition means according to the determination by the determination means, the first virtual viewpoint. It is characterized by comprising a generation means for generating an image and the second virtual viewpoint image .

According to the present invention, both the video transfer time from the actual camera and the shape estimation processing time in the image processing apparatus can be reduced. This makes it possible to generate virtual viewpoint images with high real-time characteristics.

It is a figure which shows an example of the structure of a virtual viewpoint image system. (A) is a diagram showing an example of camera arrangement, and (b) is a diagram showing the height of cameras belonging to each camera group. It is a figure which shows the shooting area of a wide-angle camera group. It is a figure which shows the shooting area of a standard camera group. It is a figure which shows the shooting area of a zoom camera group. It is a flowchart which showed the whole flow until the virtual viewpoint image is generated. It is a figure which shows an example of the GUI screen for parameter setting about a virtual camera. It is a flowchart which shows the detail of the virtual viewpoint image generation processing which concerns on Example 1. It is a figure explaining the derivation method of the shooting area of a virtual camera. It is explanatory drawing of the determination of the resolution degree of the nearest neighbor object in the provisional virtual viewpoint image. It is a flowchart which shows the flow of the shape estimation process by the billboard method which concerns on a modification. It is a figure explaining the method of specifying the object position concerning the modification. It is a figure which shows the state which the partial image of an object is projected on the flat plate which concerns on the modification. It is a flowchart which shows the flow of the process of optimizing the shooting area of the standard camera group and the zoom camera group which concerns on Example 2. FIG. It is a figure explaining how the shooting area changes for each camera group. It is a flowchart which shows the detail of the process which automatically sets various items of a virtual camera which concerns on Example 3. FIG. It is a conceptual diagram of a scene analysis process. It is a flowchart which showed the whole flow until the virtual viewpoint image is generated within the time limit which concerns on Example 4. FIG. It is a flowchart which shows the detail of the virtual viewpoint image generation processing which concerns on Example 4.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

FIG. 1 is a diagram showing an example of the configuration of a virtual viewpoint image system in this embodiment. The virtual viewpoint image is an image generated by the end user and / or an appointed operator freely manipulating the position and posture of the virtual camera, and is also called a virtual viewpoint image or an arbitrary viewpoint image. Further, the virtual viewpoint image may be a moving image or a still image. In this embodiment, an example in which the virtual viewpoint image is a moving image will be mainly described. The virtual viewpoint image system shown in FIG. 1 is composed of an image processing device 100 and three types of camera groups 109 to 111. The image processing device 100 includes a CPU 101, a main memory 102, a storage unit 103, an input unit 104, a display unit 105, and an external I / F 106, and each unit is connected via a bus 107. First, the CPU 101 is an arithmetic processing device that collectively controls the image processing device 100, and executes various programs stored in the storage unit 103 or the like to perform various processes. The main memory 102 temporarily stores data and parameters used in various processes, and also provides a work area to the CPU 101. The storage unit 103 is a large-capacity storage device that stores various data necessary for displaying various programs and GUIs (graphical user interfaces), and for example, a non-volatile memory such as a hard disk or a silicon disk is used. The input unit 104 is a device such as a keyboard, a mouse, an electronic pen, and a touch panel, and receives an operation input from a user. The display unit 105 is composed of a liquid crystal panel or the like, and performs GUI display for setting a route of a virtual camera at the time of generating a virtual viewpoint image. The external I / F unit 106 is connected to each camera constituting the camera groups 109 to 111 via LAN 108, and transmits / receives video data and control signal data. The bus 107 connects each of the above-mentioned parts and performs data transfer.

The above three types of cameras are a zoom camera group 109, a standard camera group 110, and a wide-angle camera group 111, respectively. The zoom camera group 109 is composed of a plurality of cameras equipped with a lens having a narrow angle of view (for example, 10 degrees). The standard camera group 110 is composed of a plurality of cameras equipped with a lens having a standard angle of view (for example, 30 degrees). The wide-angle camera group 111 is composed of a plurality of cameras equipped with a lens having a wide angle of view (for example, 45 degrees). Each of the cameras constituting the camera groups 109 to 111 is connected to the image processing device 100 via the LAN 108. Further, each camera group 109 to 111 starts and stops shooting, changes camera settings (shutter speed, aperture, etc.), and transfers shot video data based on a control signal from the image processing device 100.

Regarding the system configuration, there are various components other than the above, but since it is not the main subject of the present invention, the description thereof will be omitted.

FIG. 2A is a diagram showing an example of camera arrangement in an imaging system consisting of three types of cameras, a zoom camera group 109, a standard camera group 110, and a wide-angle camera group 111, in a stadium where, for example, soccer is played. be. A player as an object 202 exists on the field 201 in which the competition is performed. Then, the 12 zoom cameras 203 constituting the zoom camera group 109, the eight standard cameras 204 constituting the standard camera group 110, and the four wide-angle cameras 205 constituting the wide-angle camera group 111 surround the field 201. Have been placed. Regarding the number of cameras constituting each camera group, the relationship of zoom camera 203> standard camera 204> wide-angle camera 205 holds. Also, between the diameter length rz where the zoom camera 203 surrounds the object 202, the diameter length rs where the standard camera 204 surrounds the object 202, and the diameter length rw where the wide-angle camera 205 surrounds the object 202. , Rw> rs> rz. This is because the standard camera 204 and the wide-angle camera 205 can capture a wider area. FIG. 2B is a diagram showing the heights of the zoom camera 203, the standard camera 204, and the wide-angle camera 205 from the field 201. The relationship hw> hs> hz holds between the height hz of the zoom camera 203, the height hs of the standard camera 204, and the height hw of the wide-angle camera 205. This is also because the standard camera 204 and the wide-angle camera 205 capture a wider area.

3 to 5 are diagrams showing shooting areas of each of the camera groups 109 to 111. First, the shooting area of the wide-angle camera group 111 will be described. As shown in FIG. 3, the four wide-angle cameras 205 constituting the wide-angle camera group 111 face the wide-angle gazing point 310, which is the center of the field 201, and are evenly spaced so that the entire field 201 is within the angle of view. Is located in. At this time, the area where the shooting areas of the four wide-angle cameras 205 overlap is set as the wide-angle camera group shooting area 301, and in the area 301, the object 202 using the multi-viewpoint video data shot by the four wide-angle cameras 205. Shape estimation becomes possible. In this embodiment, an example in which the cameras are arranged at equal intervals will be mainly described, but the present invention is not limited to this. In particular, the camera layout may be decided in consideration of various circumstances such as the shape of the stadium.

Next, the shooting area of the standard camera group 110 will be described. As shown in FIG. 4, the eight standard cameras 204 constituting the standard camera group 110 are further classified into two groups A and B, group A is four standard cameras 204A, and group B is four. It consists of a standard camera 204B. The standard camera 204A of Group A faces the standard gazing point 311A and is designed to fit a specific part (left half) of the field 201 within the angle of view. Further, the standard camera 204B of the group B is designed so as to face the standard gazing point 311B and to fit a specific portion (right half) of the field 201 within the angle of view. As shown in FIG. 4, the standard cameras 204A or 204B belonging to each group are densely arranged in a direction having a high probability of shooting the front of the player, for example, and the probability of shooting the back or side of the player in other directions (for example, the probability of shooting the back or side of the player). It is sparsely arranged in the high direction of). In this way, by setting the density of camera arrangement according to the characteristics of the field, competition (event), etc., it is possible to improve the user's satisfaction with the virtual viewpoint image even with a small number of cameras, for example. However, the standard cameras may be arranged at equal intervals. Now, the area where the shooting areas of the four standard cameras 204A belonging to group A overlap is the standard camera group shooting area 311A, and the area where the shooting areas of the four standard cameras 204B belonging to group B overlap is the standard camera group shooting area. It is set to 311B. Within the standard camera group shooting area 311A, it is possible to estimate the shape of the object 202 using the multi-viewpoint video data shot by the four standard cameras 204A. Similarly, in the standard camera group shooting area 311B, it is possible to estimate the shape of the object 202 using the multi-viewpoint video data shot by the four standard cameras 204B.

Next, the shooting area of the zoom camera group 109 will be described. As shown in FIG. 5, the 16 zoom cameras 203 constituting the zoom camera group 109 are further classified into four groups C, D, E, and F. Specifically, Group C consists of four zoom cameras 203C, Group D consists of four zoom cameras 203D, Group E consists of four zoom cameras 203E, and Group F consists of four zoom cameras 203F. To. The zoom camera 203C of the group C is designed so as to face the zoom gazing point 510C and to fit a specific portion (upper left quarter) of the field 201 within the angle of view. Further, the zoom camera 203D of the group D is designed so as to face the zoom gazing point 510D and to fit a specific portion (lower left quarter) of the field 201 within the angle of view. The zoom camera 203E of the group E is designed to face the zoom gaze point 510E and to fit a specific part (upper right quarter) of the field 201 within the angle of view. The group F zoom 203F camera is designed to face the zoom gaze point 510F and fit a specific portion (lower right quarter) of the field 201 within the angle of view. As shown in FIG. 5, the zoom cameras 203C to 203F belonging to each group are densely arranged in a direction having a high probability of shooting the front of the player, and sparsely arranged in a direction having a high probability of shooting the back or side of the player. Has been done. Now, the areas where the shooting areas of the four zoom cameras 203C to 203F belonging to each group overlap are the zoom camera group shooting area 501C, the zoom camera group shooting area 501D, the zoom camera group shooting area 501E, and the zoom camera group shooting area, respectively. It is 501F. Within the zoom camera group shooting areas 501C to 501F, it is possible to estimate the shape of an object using multi-viewpoint video data shot by each of the four zoom cameras 203C to 203F.

The number and position of cameras, the number of groups, the position of the gazing point, etc. are shown as an example, and are changed according to the shooting scene and the like. For example, in this embodiment, the gazing points are the same for each group, but each camera belonging to the same group may face different gazing points at regular intervals. The interval adjustment in that case will be described in the second embodiment. Further, in the present embodiment, a camera system having three types of camera groups, that is, a zoom camera group 109, a standard camera group 110, and a wide-angle camera group 111, is described, but the present invention is not limited to this. For example, it may have only two types of camera groups, a standard camera group 110 and a wide-angle camera group 111, or it may have four or more types of camera groups. Further, in the above, an example is shown in which the number of cameras, the imaging range, and the height of installation are different for each camera group, but the number of cameras may be the same for all camera groups. The shooting range of each camera may be the same, or the height of installation of each camera may be the same. Further, factors other than the number of cameras, the shooting range, and the height of installation of each camera group may be different depending on the camera group. For example, the system may be constructed so that the number of effective pixels of the plurality of cameras belonging to the first camera group is higher than the number of effective pixels of the plurality of cameras belonging to the second camera group. Further, there may be one camera belonging to at least one camera group. As described above, the configuration of the system described in the present embodiment is only an example, and various modifications can be made according to the restrictions such as the size of the stadium, the equipment, the number of cameras, and the budget. ..

FIG. 6 is a flowchart showing the entire flow until the virtual viewpoint image is generated in the image processing device 100. This series of processes is realized by the CPU 101 reading a predetermined program from the storage unit 103, expanding it into the main memory 102, and executing this by the CPU 101.

In step 601, shooting parameters such as exposure conditions at the time of shooting and a signal for starting shooting are transmitted to each camera group 109 to 111. Each camera belonging to each camera group starts shooting according to the received shooting parameters, and holds the obtained video data in the memory in each camera.

In step 602, the multi-viewpoint video data captured by all the wide-angle cameras 205 belonging to the wide-angle camera group 111 is acquired. The acquired wide-angle video data of a plurality of viewpoints (here, four viewpoints) is expanded in the main memory 102. As described above, since the number of wide-angle cameras 205 belonging to the wide-angle camera group 111 is smaller than that of cameras belonging to other camera groups, the time required for transferring video data from each wide-angle camera group 205 can be shortened.

In step 603, the estimation process of the three-dimensional shape of the object is executed using the multi-viewpoint video data acquired from the wide-angle camera group 111. As the estimation method, a known method such as a Visual-hull method using the contour information of the object or a Multi-view stereo method using triangulation may be applied. The object area in the video data captured by the wide-angle camera 205 has a relatively low resolution. Therefore, although the 3D shape data obtained by the shape estimation in this step is low-definition and coarse, the shape of the object existing in the entire field can be estimated at high speed. The obtained object shape data is held in the main memory 102 together with the position information.

In step 604, various parameters such as the movement path of the virtual camera necessary for generating the free start point image are set based on the estimated low-definition object shape data. In this embodiment, values of various items and the like are set based on user input via a GUI (graphical user interface). 7 (a) and 7 (b) are diagrams showing an example of a GUI screen for setting parameters related to a virtual camera. On the left side of the GUI screen 700 shown in FIG. 7A, a wide-angle camera group shooting area 301 is displayed on a bird's-eye view (field map 701) of the entire shooting space including the field 201. The three-dimensional shape 702 of the object acquired in step 603 is mapped on the wide-angle camera group shooting area 301. The user can confirm the position of the object 202, the direction in which the object 202 is facing, and the like by the mapped object three-dimensional shape 702. The user presses the virtual camera path setting button (not shown) and then operates the mouse or the like on the wide-angle camera group shooting area 301 to move the cursor 703, thereby designating the movement trajectory as the virtual camera path 704. can do. The height of the virtual camera path specified at this time from the field 201 is a default value (for example, 15 m). Then, after designating the virtual camera path 704, the user can press the height edit button (not shown) to change the height of the designated virtual camera path. Specifically, the virtual whose altitude is to be changed by moving the cursor 703 to an arbitrary position (coordinates) on the virtual camera path displayed on the wide-angle camera group shooting area 301 and performing a click operation with a mouse or the like. Specify the camera position (height edit point). Now, the part indicated by the cross in the wide-angle camera group shooting area 301 indicates the height edit point specified by the user. It is possible to set a plurality of height edit points. In the example of FIG. 7A, two height edit points P1 and P2 are set. When the height edit point is set, the height setting window 705 is displayed on the right side of the GUI screen 700. The user can change the height of the virtual camera at the position by inputting an arbitrary value (unit: m) in the input field 706 corresponding to each editing point in the height setting window 705. In this case, the height other than the place where the altitude is changed by the height editing point is interpolated from the height editing point at the adjacent position or the default value, and the height is adjusted so as not to change suddenly. The user who has specified the virtual camera path then presses the time frame setting button (not shown) to set the time (moving speed) required for the virtual camera to pass through the virtual camera path. Specifically, in response to pressing the time frame setting button, the time frame setting window 707 is displayed on the right side of the GUI screen 700, and the time required to move to the input field 708 (item: t), the input field 709 (item). : Fps) Enter each value of the frame rate. When the time and frame rate are input, the number of frames of the generated virtual viewpoint image is calculated and displayed in the display field 710 (item: frame). In the example of FIG. 7A, the time input to the input field 708 is 2 [s] and the frame rate input to the input field 709 is 60 [fps], so that it is viewed from a virtual viewpoint for 120 frames. An image (hereinafter referred to as a virtual viewpoint image) will be generated. The number of frames calculated at this time is held in the main memory 102 as “F_Max”. Further, the user presses the gaze point setting button (not shown) to set the gaze point position of the virtual camera in order to determine the direction in which the virtual camera faces on the designated virtual camera path. Specifically, the gazing point is set by moving the cursor 703 to an arbitrary position (coordinates) on the virtual camera path displayed in the wide-angle camera group shooting area 301 and performing a click operation with a mouse or the like. Specify the target virtual camera position (gaze point setting point). As with the height edit points, it is possible to set multiple gaze point setting points. When the gaze point setting points are set, the current gaze point position paired with them is automatically displayed. The gazing point position at this time is a predetermined position of the object of interest such as a player holding the ball. In FIG. 7B, the points indicated by Δ are the gaze point setting points (virtual camera positions) designated by the user, and the points indicated by ☆ are the corresponding gaze points. In the example of FIG. 7B, two gaze point setting points C1 and C2 are set, and T1 is displayed as the gaze point corresponding to C1 and T2 is displayed as the gaze point corresponding to C2. When the gaze point setting point is set, the gaze point setting window 711 is displayed on the right side of the GUI screen 700. The user can change the position of the virtual camera at the gaze point setting point by inputting arbitrary coordinates (x, y, z) in the input field 712 corresponding to each setting point in the gaze point setting window 711. can. Then, the gazing points other than the place where the gazing point is changed are interpolated from the gazing point setting point at the adjacent position or the default gazing point, and adjusted so that the gazing point does not change suddenly. As described above, the parameters related to the virtual camera are set.

In step 605, in order to generate virtual viewpoint images for the number of frames set in step 604, a storage area for the variable F is secured in the main memory 102, and “0” is set as an initial value. Then, in the following step 606, the virtual viewpoint image of the F frame is generated according to the set virtual camera parameters. The details of the virtual viewpoint image generation process will be described in detail later.

In step 607, the value of the variable F is incremented (+1). Then, in step 608, it is determined whether or not the value of the variable F is larger than the above-mentioned F_Max. As a result of the judgment, if the value of the variable F is larger than F_Max, it means that the virtual viewpoint images corresponding to the set number of frames have been generated (that is, the virtual viewpoint image corresponding to the set time frame is completed). Therefore, the process proceeds to step 609. On the other hand, when the value of the variable F is F_Max or less, the process returns to step 606 and the virtual viewpoint image generation process of the next frame is executed.

In step 609, it is determined whether or not to change the setting of the virtual camera parameter to generate a new virtual viewpoint image. This process is performed based on an instruction from a user who has confirmed the image quality and the like by looking at the virtual viewpoint image displayed in the preview window 713 displayed when the preview button (not shown) is pressed. When the user wants to regenerate the virtual viewpoint image, he / she presses the virtual camera path setting button or the like again, and sets the parameters related to the virtual camera again (returns to step 604). Then, a virtual viewpoint image is generated with the contents according to the newly set virtual camera parameters. On the other hand, if there is no problem with the generated virtual viewpoint image, this process ends. The above is a rough flow until the virtual viewpoint image is generated according to this embodiment.

Subsequently, the virtual viewpoint image generation process in step 606 described above will be described in detail. FIG. 8 is a flowchart showing the details of the virtual viewpoint image generation process according to the present embodiment. Hereinafter, a detailed description will be given along with the flow of FIG.

In step 801 are acquired the virtual camera position and the gazing point position in the frame Fi of interest to be processed, respectively, based on the virtual camera path set in step 605 described above. In the following step 802, the shooting area Vr of the virtual camera of the frame Fi of interest is derived from the acquired virtual camera position and gaze point position. FIG. 9 is a diagram illustrating a method of deriving a shooting area of a virtual camera. In FIG. 9, a quadrangular pyramid is formed from the virtual camera 901 toward the gazing point 902, and the rectangular region 903 that is the intersection of the quadrangular pyramid and the field 201 becomes the virtual camera shooting region Vr. Then, in step 803, the object closest to the gazing point position acquired in step 801 is detected and set as the nearest neighbor object. In FIG. 9, reference numeral 904 indicates the nearest neighbor object.

In step 804, the resolution of the nearest neighbor object in the set virtual camera is calculated. Specifically, the ratio R of the area occupied by the nearest neighbor object in the provisional virtual viewpoint image (virtual viewpoint image based only on the multi-viewpoint video data of the wide-angle camera 205) seen from the virtual camera of the frame Fi of interest is obtained. This ratio R is a value obtained by dividing the number of pixels of the nearest neighbor object area in the provisional virtual viewpoint image by the total number of pixels of the entire image, and is a value in the range of 0 to 1 such as 0.3. .. In this embodiment, an example of evaluating a provisional virtual viewpoint image based on the resolution of the nearest neighbor object will be mainly described, but it is different in addition to the nearest neighbor object or in place of the nearest neighbor object. You may want to evaluate the resolution of the object. Examples of other objects are, for example, the object selected by the viewer (eg, a particular player), the object closest to the center of the provisional virtual viewpoint image, and the object facing forward (if more than one). Among them, the object closest to the virtual camera) and so on. Further, the number of objects referred to for the evaluation of the provisional virtual viewpoint image is not limited to one, and may be plural.

In step 805, it is determined whether or not the nearest neighbor object exists in the standard camera group shooting area based on the respective position coordinates. In this case, the position information of the nearest object is derived in step 603 described above and is stored in the RAM main memory 102, and the position information of the standard camera group shooting area is stored in the storage unit 103 in advance. .. If the nearest neighbor object is in the standard camera group shooting area, the process proceeds to step 806. On the other hand, if it does not exist, the process proceeds to step 813, and rendering using low-definition object shape data based on the multi-viewpoint video data of the wide-angle camera group is executed. In the case of this embodiment, if the nearest neighbor object is included in either the standard camera group shooting area A or B, the process proceeds to step 806.

In step 806, it is determined whether or not the ratio R representing the resolution of the nearest neighbor object in the provisional virtual viewpoint image is larger than the first threshold value Rs. Here, the first threshold Rs is a pixel of the nearest object region in the captured image obtained by acquiring an image captured by any of the standard cameras 204 belonging to the standard camera group determined to include the nearest object. It is obtained by dividing the number by the total number of pixels. This makes it possible to compare the resolution of the nearest neighbor object between the virtual camera of the frame Fi of interest and the standard camera. FIG. 10A is a diagram that visually represents the determination content in this step. In this case, the resolution of the nearest neighbor object in the provisional virtual viewpoint image is higher (the value of the ratio R is larger). Will be determined. As a result of the determination, if the calculated value of the ratio R is larger than the threshold value Rs, the process proceeds to step 807. On the other hand, when the value of the calculated ratio R is equal to or less than the threshold value Rs, the process proceeds to step 813, and rendering is executed using the low-definition object shape data generated based on the multi-viewpoint video data of the wide-angle camera group. There are various modifications in the determination method of step 806. For example, if the ratio R is larger than the threshold value Rs by a predetermined threshold value or more, the process may proceed to step 807, and if not, the process may proceed to step 813.

In step 807, as in step 805 described above, it is determined whether or not the nearest neighbor object exists in the zoom camera group shooting area based on the respective position coordinates. In this case, the position information of the zoom camera group shooting area is also stored in the storage unit 103 in advance. If the nearest neighbor object exists in the zoom camera group shooting area, the process proceeds to step 808, and if it does not exist, the process proceeds to step 810. In the case of this embodiment, if the nearest neighbor object is included in any of the zoom group shooting areas C to F, the process proceeds to step 808.

In step 808, it is determined whether or not the ratio R representing the resolution of the nearest neighbor object in the provisional virtual viewpoint image is larger than the second threshold value Rz. Here, the second threshold Rz is the number of pixels of the nearest object region in the captured image obtained by acquiring the captured image of any of the zoom cameras 203 belonging to the zoom camera group determined to include the nearest object. Is obtained by dividing by the total number of pixels. This makes it possible to compare the resolution of the nearest neighbor object between the virtual camera of the frame Fi of interest and the zoom camera. FIG. 10B is a diagram that visually expresses the determination content in this step, and again, it is said that the resolution of the nearest neighbor object in the provisional virtual viewpoint image is higher (the value of the ratio R is larger). It will be judged. As a result of the determination, if the calculated value of the ratio R is larger than the threshold value Rz, the process proceeds to step 809. On the other hand, if the calculated value of the ratio R is equal to or less than the threshold value Rz, the process proceeds to step 810.

In step 809, the multi-viewpoint video data used for high-definition estimation (re-estimation) of the object shape in the virtual camera shooting area Vr of the frame Fi of interest corresponds to the zoom camera group shooting area where it is determined that the nearest object exists. Obtained from the zoom camera group. The acquired multi-viewpoint video data is expanded in the main memory 102. Further, in step 810, the multi-viewpoint video data used for re-estimating the object shape (high definition) in the virtual camera shooting area Vr of the frame Fi of interest corresponds to the standard camera group shooting area where it is determined that the nearest object exists. Obtained from the standard camera group. The acquired multi-viewpoint video data is expanded in the main memory 102.

In step 811, the object shape re-estimation process is executed using the multi-viewpoint video data expanded in the main memory 102. As a result, higher-definition object shape data than the object shape data obtained in step 603 described above is acquired. Then, in step 812, the low-definition object shape data obtained by the shape estimation in step 603 is replaced with the high-definition object shape data obtained by the shape estimation in step 811.

In step 813, a virtual viewpoint image which is an image seen from the virtual camera of the frame Fi of interest is generated by using the object shape data determined by the processes up to step 812 and the rendering method in computer graphics.

The above is the content of the virtual viewpoint image generation process according to this embodiment. In this embodiment, the resolution of the nearest neighbor object in the provisional virtual viewpoint image is used as an index in determining whether to re-estimate the object shape and acquire higher-definition object shape data. Not limited to this. For example, using the distance between the nearest object and the virtual camera as an index, re-estimation is performed when the distance between the nearest object and the virtual camera position is farther than the distance between the nearest object and the zoom camera position or the standard camera position. You may do it. Further, in the above-described embodiment, the resolution of the provisional virtual viewpoint image (specifically, the ratio R obtained by dividing the number of pixels of the nearest object by the number of pixels of the provisional virtual viewpoint image) and Based on the comparison result with the threshold value (specifically, the threshold Rs obtained by dividing the number of pixels of the nearest object in the captured image obtained by a camera belonging to the standard camera group by the number of pixels of the captured image). The explanation focused on an example of determining whether or not to generate a high-quality virtual viewpoint image. However, the present invention is not limited to this determination method, and various modifications are possible. For example, if the ratio R in the provisional virtual viewpoint image is larger than a predetermined threshold value (that is, if the size of the object in the virtual viewpoint image is larger than the threshold value), a high-quality virtual viewpoint image should be generated regardless of the threshold value Rs. It may be determined that. Alternatively, as another method, the image quality of the nearest neighbor object in the provisional virtual viewpoint image may be evaluated, and it may be determined whether or not a high quality virtual viewpoint image should be generated according to the evaluation result. As a method of evaluating the image quality of the nearest neighbor object, for example, if the object is a human facing the front, a method of evaluating based on the recognition result of the face may be used, or based on the sharpness of the edge of the object. The evaluation method may be used. If these determination methods are used, it is possible to make a simpler determination than the determination method using the captured image of the standard camera group. Other modifications will be described below.

<Modification example>
In the above embodiment, the low-definition object 3D shape is first acquired using the multi-viewpoint video data of the wide-angle camera group, and then the multi-viewpoint video data of the standard or zoom camera group is used according to the virtual camera path. The high-definition object 3D shape was reacquired to generate a virtual viewpoint image. However, it is not limited to this. For example, instead of the low-definition three-dimensional shape estimation using the multi-viewpoint video data of the wide-angle camera group, the two-dimensional shape estimation in which the object is regarded as a plane may be performed (Billboard method). In the case of the billboard method, the flow shown in FIG. 11 is executed in step 603 described above. Hereinafter, it will be described in detail.

In step 1101, the object position on the field 201 is specified. FIG. 12 is a diagram illustrating a method of specifying an object position. In FIG. 12, the wide-angle camera image _1 of (a) and the wide-angle camera image _2 of (b) are images taken by different wide-angle cameras 205, and one line 1201 and an object 1202 are shown in each image. ing. FIG. 12 (c) is a composite image after the projection transformation, which is a composite image obtained by performing a projective transformation on the wide-angle camera image _1 and the wide-angle camera image _2 with the field plane as a reference. In the composite image after the projective transformation, it can be seen that the line 1201 remains one line, but the object 1202 is separated into two lines. Utilizing this characteristic, the position indicated by the cross, which is the base point of separation, is specified as the object position 1203.

In step 1102, a flat plate is installed at the specified object position. Then, in the following step 1103, a partial image of the object cut out from the captured image of the wide-angle camera 205 is projected onto the installed flat plate. FIG. 13 is a diagram showing a state in which a partial image of an object is projected on a flat plate. It can be seen that the partial image of each object is projected on the flat plate 1300 installed as many as the number of objects existing on the field 201.

The above is the content of the processing related to this modification. Since the shape of the object is processed in two dimensions, high-speed processing is possible.

Also, instead of installing a flat plate and projecting a cutout image, a prepared object shape (for example, a scan with a 3D range scanner or a manually modeled object shape) can be placed at the specified object position. good.

In the case of this modification, a part of the processing in the virtual viewpoint image generation processing (step 606) will be changed. That is, even when the determination process of step 805 and step 806 is "No", the process proceeds to step 811 instead of step 813 to execute the estimation process of the three-dimensional shape of the object. For the estimation at this time, the multi-viewpoint video data of the wide-angle camera group acquired in step 602 will be used. The dashed arrow 800 in the flow of FIG. 8 indicates this. In this case as well, the processing speed can be increased by estimating the shape of only the objects included in the shooting area Vr of the virtual camera of the frame Fi of interest.

As described above, according to this embodiment, only when the virtual camera can be closer to the object of interest while maintaining the image quality, the multi-viewpoint video data taken by the camera group with a narrower angle of view is acquired and the high-definition object is obtained. Performs shape estimation and virtual viewpoint image generation. Therefore, the minimum required data transfer amount and processing load can be suppressed. This makes it possible to generate a virtual viewpoint image with higher real-time performance.

Next, it is possible to further reduce the video transfer time and shape estimation processing time by optimizing the shooting area of the camera group other than the wide-angle camera group (zoom camera group and standard camera group in Example 1) according to the shooting scene. The embodiment of the above will be described as Example 2. Since the general process of system configuration, virtual viewpoint image generation, and processing is the same as that of the first embodiment, the description thereof will be omitted, and the differences will be briefly described below.

FIG. 14 is a flowchart showing a flow of processing for optimizing the shooting area of the standard camera group and the zoom camera group according to the present embodiment. As a premise of this embodiment, in the default state before executing this process, each camera belonging to the same group in each camera group faces in a different direction at regular intervals (the gazing point is different). And.

In step 1401, the shooting scene is set (for example, whether the shooting target is a ball game or an athletics target) based on a user input via a UI screen (not shown). In the following step 1402, it is determined whether or not the set shooting scene is a scene that needs to shoot a high altitude region equal to or higher than a predetermined altitude. Here, the shooting scene that requires shooting in a high altitude region is a ball game such as soccer or rugby in which the ball reaches several tens of meters. Further, a shooting scene below a predetermined altitude and not requiring shooting in a high altitude region (a scene in which shooting in a low altitude region is sufficient) is a short-distance run on land. As a result of the determination, if the shooting scene involves shooting in a high altitude region, the process proceeds to step 1403. On the other hand, in the case of a shooting scene that does not involve shooting in a high altitude region, the process proceeds to step 1404.

In step 1403, the positions of the cameras belonging to the zoom camera group 109 and the standard camera group 110 remain fixed, and the distance between the gazing points is reduced in group units (or the gazing points of the cameras in the group are made the same). ). Further, in step 1404, the distance between the gazing points is expanded (or maintained) in group units while maintaining the positions of the cameras belonging to the zoom camera group 109 and the standard camera group 110. FIG. 15 is a diagram illustrating how the standard camera group shooting area and the zoom camera group shooting area change by adjusting the distance between the gazing points in units of each camera group. FIG. 15A is an explanatory diagram when the distance between gazing points is reduced to the maximum (changed to the same gazing point). In FIG. 15A, the two white circles 1501 and 1502 indicate the respective gazing points of the two cameras 1511 and 1512 before the change. One black circle 1503 indicates the gaze point after the change, and the two cameras 1511 and 1512 both point to the same gaze point. At this time, the camera group shooting area X along the field surface becomes narrower, but the camera group shooting area Z in the height direction becomes wider than before the change. Therefore, it is a shooting area suitable for shooting a ball game or the like where the ball reaches a high altitude. On the other hand, FIG. 15B is an explanatory diagram when the distance between the gazing points is increased. In FIG. 15B, the black circle 1504 shows the gaze point after the change of the camera 1511, and the black circle 1505 shows the gaze point after the change of the camera 1512, and it can be seen that the interval between the gaze points is widened. .. At this time, the camera group shooting area X along the field surface becomes wide, but the camera group shooting area Z in the height direction becomes narrow. Therefore, in short-distance running on land, it is possible to photograph a wide area parallel to the field surface.

The above is the content of the process for optimizing the shooting area of the standard camera group and the zoom camera group according to this embodiment. In this embodiment, the distance between gaze points is reduced when the altitude is higher than the predetermined altitude, and the distance between gaze points is increased (or maintained) when the altitude is lower than the predetermined altitude, using a single predetermined altitude as a reference (threshold value). However, it is not limited to this. For example, a threshold value for reducing the distance between gazing points and a threshold value for increasing the distance between gaze points may be provided separately. By this processing, the number of cameras required for shooting one competition can be reduced. In addition, it can be expected to improve convenience, such as shooting other competitions at the same time using a reduced number of cameras.

According to this embodiment, the shooting area of the camera group other than the wide-angle camera group can be optimized according to the shooting scene. This makes it possible to further reduce the video transfer time and processing time.

Subsequently, a mode in which the setting related to the virtual camera is automatically performed using the database will be described as Example 3. The contents common to the first and second embodiments will be omitted, and the differences will be mainly described below.

FIG. 16 is a flowchart showing the details of the process for automatically setting various items of the virtual camera, which is executed in place of step 604 in the flow of FIG. 6 described above, according to this embodiment.

In step 1601, a shooting scene analysis database (hereinafter referred to as “scene DB”) connected via an external network (not shown) is requested to analyze the shooting scene. The image processing device 100 is connected to the scene DB via LAN 108, and the scene DB is further installed on a network that can be connected from the outside. The scene DB stores various information about past shooting scenes, receives information necessary for analysis from the image processing device 100, and executes analysis processing of the shooting scene. FIG. 17 is a conceptual diagram of the scene analysis process. In the scene DB 1700, object transition information and shooting environment information are recorded for each type of shooting scene. Here, the object transition information includes, for example, when the shooting scene is a game of sports competition, data recording the locus of the movement position of the player, data recording the locus of the change of the player shape, and further, in the case of a ball game, the position of the ball. It includes data that records the trajectory of movement. The shooting environment information is data that records the surrounding environment at the time of shooting, for example, the sound of the audience seats. In a decisive scene in a sports competition, the volume in the audience seats increases due to cheers, so that it can be used to determine whether or not the decisive scene is of great interest to the viewer. Further, when the shooting scene is a game of a sports competition, the scene DB1700 also contains information indicating the correspondence between the above-mentioned object transition information and shooting environment information and the decisive scene in each competition (hereinafter, decisive scene information). It has been recorded. The decisive scene information is composed of the type of the decisive scene and the typical camera work (movement path of the virtual camera) suitable for the decisive scene. The types of decisive scenes are, for example, in soccer, a shooting scene, a long pass scene, a corner kick scene, and the like. Definitive scene information is retained as learning data, and it is possible to analyze the shooting scene by using deep learning technology or the like. Since the material of the learning data can be obtained from stadiums around the world via the Internet or the like, a huge amount of data can be collected. The image processing device 100 transmits and analyzes the type of shooting scene (competition), the movement log of the player or ball (movement trajectory data), the shape log of the player (shape change data), and the audio data of the spectators' seats to the scene DB 1700. To ask. The data to be transmitted to the scene DB 1700 is generated based on the multi-viewpoint video data of the wide-angle camera group 111 acquired in step 602. In the scene DB 1700, the above-mentioned analysis process is performed in response to the analysis request. The analysis result is sent to the image processing apparatus 100.

In step 1602, the image processing apparatus 100 receives the analysis result from the scene DB 1700. The analysis result includes information on the position where the decisive scene occurred, the type of the decisive scene, and the representative camera work suitable for the decisive scene.

In step 1603, various items of the virtual camera are automatically set based on the received analysis result. Specifically, the position where the decisive scene is generated is set as the gazing point of the virtual camera. In addition, the movement path of the virtual camera and the corresponding time frame are set based on the typical camera work. Information indicating the type of the decisive scene is added as metadata to the generated virtual viewpoint image. This metadata is referred to for secondary use by broadcasters (character effect insertion, database creation, etc.).

The above is the content of the process for automatically setting various items of the virtual camera. The gaze point and the movement route of the virtual camera automatically set as described above may be displayed on the GUI screen 700 described above, and the contents may be edited by the user. Further, in this embodiment, the scene DB 1700 is configured as a device separate from the image processing device 100, but both may be integrated into one device. Alternatively, the scene analysis function and the data storage function of the scene DB1700 of this embodiment may be separated and each may be configured by a separate device.

According to this embodiment, various items such as a movement route for a virtual camera can be automatically set by using a database. This makes it possible to further reduce the processing time.

In this embodiment, a video generation method that is particularly suitable when the generation time of the virtual viewpoint image is limited will be described. As a case where the generation time is limited, for example, there is a case where a virtual viewpoint image is generated as a replay immediately after play, or a case where a virtual viewpoint image is generated in real time during sports broadcasting. The description of the process overlapping with the first embodiment will be omitted.

FIG. 18 is a flowchart showing the entire flow until the virtual viewpoint image is generated within the time limit in the image processing apparatus 100. This series of processes is realized by the CPU 101 reading a predetermined program from the storage unit 103, expanding it into the main memory 102, and executing this by the CPU 101.

Steps 1801 to 1809 are substantially the same as steps 601 to 609. The difference from FIG. 6 is step 1806 and step 1810. After the multi-viewpoint video data is acquired from the wide-angle camera group in step 1802, step 1810 is executed in parallel with steps 1803 to 1805. In step 1810, the multi-viewpoint video data of the standard camera group is sequentially acquired in order to effectively utilize the communication band of the LAN 108 for which communication has been completed. The acquired multi-viewpoint video data is used in step 1806. The virtual viewpoint image generation process in step 1806 will be described in detail with reference to FIG.

The above is a rough flow until the virtual viewpoint image is generated according to this embodiment. By parallelizing data communication, which has a long processing time, with shape estimation processing and virtual camera parameter setting processing, there is an effect that the overall processing time is significantly reduced. The configuration of this embodiment may be used in a case where the generation time of the virtual viewpoint image is not limited or a case where the generation time has a sufficient margin.

The virtual viewpoint image generation process of step 1806 of FIG. 18 will be described. Here, after the object shape is generated using the multi-viewpoint video data of the standard camera group, the multi-viewpoint video data of the zoom camera group is applied in consideration of the processing time. FIG. 19 is a flowchart showing the details of the virtual viewpoint image generation process according to this embodiment. Hereinafter, a detailed description will be given along with the flow of FIG.

Since step 1901, step 1906 is the same process as steps 801 to 806, the description thereof will be omitted. In step 1907, the object shape re-estimation process is executed using the multi-viewpoint video data of the standard camera group acquired in step 1810. In step 1908, the low-definition object shape data obtained by the shape estimation in step 603 described above is replaced with the high-definition object shape data obtained by the shape estimation in step 1907. Steps 1909 and 1910 are the same processes as steps 807 and 808. In step 1911, it is determined whether the processing time up to this step is within the limit value for performing shape estimation. The limit value is predetermined based on the time during which the shape estimation of one image frame must be performed. For example, when generating 600 frames of video for replay within 30 seconds, the limit value can be 50 milliseconds (30,000/600) per image frame. However, the limit value may be different depending on the processing time and other circumstances. If Yes, the process proceeds to step 1912. If No, the process proceeds to step 1917 to generate a virtual viewpoint image using a standard camera group. That is, if No is determined in step 1911, the shape is not estimated based on the multi-viewpoint video data of the zoom camera group regardless of the evaluation result of the object shape data replaced in step 1908. In step 1912, the plurality of viewpoint video data is acquired from the zoom camera group as in step 809. At this time, in order to secure the communication band of the LAN 108, the acquisition of the multi-viewpoint video data of the standard camera group in step 1810 is suspended and restarted after the end of this step. In step 1913, it is determined again whether the processing time up to this step is within the limit value for performing shape estimation. If Yes, the process proceeds to step 1914, and if No, the process proceeds to step 1916. In step 1914, the object shape re-estimation process is executed using the multi-viewpoint video data of the zoom camera group. In step 1915, the object shape data obtained by the shape estimation in step 1907 is replaced with the high-definition object shape data obtained by the shape estimation in step 1914. In step 1916, since the re-execution time of the shape estimation is insufficient, the data obtained in step 1907 is used for the object shape, but the texture projected on the object shape uses the multi-viewpoint video data of the zoom camera. Rendering settings are made. In step 1917, a virtual viewpoint image, which is an image seen from the virtual camera of the frame Fi of interest, is generated using the object shape and texture determined by the processes up to step 1916.

The timing for executing the determination of whether or not the processing time is within the limit value of shape estimation is not limited to the example shown in FIG. For example, the determination may be made between steps 1906 and 1907, or the determination may be made between steps 1908 and 1909. Further, the order of steps 1910 and 1911 in FIG. 19 may be reversed.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (15)

It is a plurality of captured images obtained by a plurality of first imaging devices that capture fields from different directions, and is used to generate a first virtual viewpoint image according to the position of a virtual viewpoint and the line-of-sight direction from the virtual viewpoint. The first acquisition means for acquiring a plurality of captured images ,
A second virtual viewpoint image having a higher image quality than the first virtual viewpoint image is obtained based on one or more shot images obtained by one or a plurality of second shooting devices that shoot at least a part of the field from different directions. A determination means for determining whether or not to generate according to the evaluation result of the first virtual viewpoint image generated based on the plurality of captured images acquired by the first acquisition means .
A second acquisition means for acquiring one or a plurality of captured images obtained by the second imaging device according to the determination by the determination means.
The first virtual viewpoint image and the first virtual viewpoint image based on the plurality of captured images acquired by the first acquisition means and one or a plurality of captured images acquired by the second acquisition means according to the determination by the determination means. The generation means for generating the second virtual viewpoint image and
A virtual viewpoint image generator characterized by being equipped with. The virtual viewpoint image generation device according to claim 1, wherein the determination means determines whether or not to generate the second virtual viewpoint image based on the evaluation result of the image quality of the object included in the first virtual viewpoint image. .. The determination means is
When the size of the object in the first virtual viewpoint image is equal to or larger than the threshold value, it is determined to generate the second virtual viewpoint image.
The virtual viewpoint image generation device according to claim 1 or 2 , wherein when the size of the object is less than the threshold value, it is determined not to generate the second virtual viewpoint image. The determination means determines whether or not to generate the second virtual viewpoint image based on the evaluation result of the image quality of the object closest to the virtual viewpoint among the plurality of objects included in the first virtual viewpoint image. The virtual viewpoint image generation device according to any one of claims 1 to 3, wherein the virtual viewpoint image generation device is characterized. The determination means is
The image quality of the first virtual viewpoint image generated based on the plurality of captured images obtained by the plurality of first photographing devices , and the one or more captured images obtained by the one or the plurality of second photographing devices . Compare with the threshold for image quality based on
The invention according to any one of claims 1 to 4 , wherein when the image quality of the first virtual viewpoint image is lower than the image quality represented by the threshold value, it is determined to generate the second virtual viewpoint image. The virtual viewpoint image generator described. The number of the plurality of first photographing devices is smaller than the number of the one or more second photographing devices .
The virtual viewpoint image generation device according to any one of claims 1 to 5 , wherein the shooting range of each of the plurality of first shooting devices is wider than the shooting range of the second shooting device. The determination means includes a first parameter determined based on the ratio of the number of pixels of a predetermined object included in the first virtual viewpoint image to the number of pixels of the first virtual viewpoint image, and the plurality of first photographing devices . The first is determined based on the ratio of the number of pixels of the predetermined object in the plurality of captured images obtained by the above to the number of pixels of the predetermined object in the one or a plurality of captured images obtained by the one or a plurality of second photographing devices . The virtual viewpoint image generation device according to any one of claims 1 to 6 , wherein it is determined whether or not to generate the second virtual viewpoint image based on a comparison with two parameters. A third virtual viewpoint image having a higher image quality than the second virtual viewpoint image generated in response to a determination by the determination means is generated based on one or a plurality of captured images obtained by one or a plurality of third photographing devices . A second determination means for determining whether or not to do so according to the evaluation result of the second virtual viewpoint image.
A third acquisition means for acquiring one or a plurality of captured images obtained by the third imaging device according to the determination by the second determination means.
Further have
The generation means generates a third virtual viewpoint image based on one or a plurality of captured images acquired by the third acquisition means in response to a determination by the second determination means.
The virtual viewpoint image generation device according to any one of claims 1 to 7 . The second determination means is
When the second virtual viewpoint image is generated within a predetermined time limit, it is determined whether or not to generate the third virtual viewpoint image according to the evaluation result of the second virtual viewpoint image.
If the second virtual viewpoint image is not generated within the predetermined time limit, it is determined not to generate the third virtual viewpoint image regardless of the evaluation result of the second virtual viewpoint image. The virtual viewpoint image generation device according to claim 8. The virtual according to any one of claims 1 to 9 , wherein the determination means determines whether or not to generate the second virtual viewpoint image based on a predetermined time limit. Viewpoint image generator. The determination means is
When the first virtual viewpoint image is generated within the predetermined time limit, it is determined whether or not to generate the second virtual viewpoint image according to the evaluation result of the first virtual viewpoint image.
If the first virtual viewpoint image is not generated within the predetermined time limit, it is determined not to generate the second virtual viewpoint image regardless of the evaluation result of the first virtual viewpoint image. 10. The virtual viewpoint image generation device according to claim 10 . It is used to generate a first virtual viewpoint image according to the position of a virtual viewpoint and the line-of-sight direction from the virtual viewpoint based on a plurality of captured images obtained by a plurality of first imaging devices that capture fields from different directions. The first acquisition process to acquire multiple captured images ,
Whether to generate a second virtual viewpoint image having a higher image quality than the first virtual viewpoint image based on the captured images obtained by one or a plurality of second photographing devices that capture at least a part of the field from different directions. A determination step of determining whether or not the image is based on the evaluation result of the first virtual viewpoint image generated based on the plurality of captured images acquired in the first acquisition step.
A second acquisition step of acquiring one or a plurality of captured images obtained by the second imaging device according to the determination, and a second acquisition step.
The first virtual viewpoint based on a plurality of captured images acquired in the first acquisition step and one or a plurality of captured images acquired in the second acquisition step according to a decision in the determination step. A generation process for generating an image and the second virtual viewpoint image, and
A virtual viewpoint image generation method characterized by including. The deciding step is characterized in that it is determined whether or not to generate the second virtual viewpoint image based on the evaluation result of the image quality of the object included in the first virtual viewpoint image. The described virtual viewpoint image generation method. In the determination step, when the size of the object in the first virtual viewpoint image is equal to or larger than the threshold value, it is determined to generate the second virtual viewpoint image, and when the size of the object is smaller than the threshold value, it is determined. The method for generating a virtual viewpoint image according to claim 12 , wherein it is determined not to generate the second virtual viewpoint image. A program for causing a computer to function as the virtual viewpoint image generator according to any one of claims 1 to 11 .

Images