JP6759375B2 - Systems, methods and programs that generate virtual viewpoint images - Google Patents

Description

The present invention relates to a technique for generating an image from a virtual viewpoint based on a plurality of images captured from a plurality of viewpoint positions.

Recently, a virtual viewpoint image technology that reproduces an image from a camera (virtual camera) that does not actually exist and is virtually arranged in a three-dimensional space by using images taken by a plurality of real cameras is attracting attention. .. According to this virtual viewpoint image technology, for example, highlight scenes in soccer and basketball games can be viewed from various angles, so that a user can be given a higher sense of presence.

In order to generate a virtual viewpoint image, it may be necessary to aggregate image data taken by a plurality of real cameras into an image processing server or the like, and to generate or render a three-dimensional model (object shape data) on the server or the like.

As a method for estimating the three-dimensional shape of an object, a method called "visual volume crossing method (Visual Hull)" is known (Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-10805

In the conventional technique for estimating the three-dimensional shape, there is a possibility that a three-dimensional model may not be generated for a structure that is a stationary object such as a soccer goal existing in the shooting range. This is because the object for which the three-dimensional shape is estimated is the foreground part of a moving object such as a human being in the captured image. That is, a stationary structure such as a soccer goal is treated as a background and is not a target for generating a three-dimensional model. When a virtual viewpoint image is generated when a three-dimensional model of a structure is not generated, a non-moving structure or the like is represented two-dimensionally behind a moving person or the like, as if it were stuck to the ground or the like. It will be expressed, and the image expression will be far from the actual shooting scene. An example is shown in FIG. FIG. 1 is a virtual viewpoint image of a soccer scene, which looks like a soccer goal (all elements of a goal post, a crossbar, and a goal net) stuck to a grass field. Further, FIG. 13B is a virtual viewpoint image of one scene of sumo wrestling, but it is an image as if a sumo wrestler who should have been pushed out and fell under the ring is lying on the ring.

The present invention has been made in view of the above problems, and an object of the present invention is to obtain a natural virtual viewpoint image.

The system according to the present invention acquires the first generation means for generating three-dimensional shape data corresponding to an object photographed from a plurality of directions and the three-dimensional shape data corresponding to a structure photographed from a plurality of directions. A first acquisition means, a second acquisition means for acquiring background data corresponding to a background different from at least the object and the structure, which are photographed from a plurality of directions, and a second acquisition means for acquiring information indicating a designated viewpoint. The 3 acquisition means, the 3D shape data corresponding to the object generated by the 1st acquisition means, the 3D shape data corresponding to the structure acquired by the 1st acquisition means, and the 2nd acquisition. The first generation means includes a second generation means for generating an image based on the background data acquired by the means and the information indicating the viewpoint acquired by the third acquisition means. Corresponds to the object based on an image that includes the area of the object, the area of the structure, and the background area, and an image that does not include the area of the object but includes the area of the structure and the background area. It is characterized in that it can generate three-dimensional shape data.

According to the present invention, a natural virtual viewpoint image can be obtained.

Diagram explaining the problems of the conventional method The figure which shows an example of the arrangement of the camera system which concerns on Embodiment 1. Diagram showing an example of the hardware configuration of the virtual viewpoint image generation system The figure explaining the common shooting area of a plurality of cameras Explanatory diagram of volume data A sequence diagram showing a structure model generation process according to the first embodiment. (A) shows a photographed image of a field without a soccer goal, and (b) shows a captured image of a field with a soccer goal. Figure showing 3D model of soccer goal on volume data A sequence diagram showing a process of generating a virtual viewpoint image according to the first embodiment. (A) shows an example of a captured image, (b) shows an example of a foreground image, and (c) shows an example of a virtual viewpoint image. Figure showing 3D model of player on volume data The figure which shows an example of the arrangement of the camera system which concerns on the modification of Embodiment 1. (A) and (b) are diagrams for explaining the problems of the conventional method. A diagram showing that the surrounding area is divided into four areas on a bird's-eye view of the ring from directly above. A flowchart showing a flow of processing for thinning out and transmitting image data of a structure portion in a shooting scene according to the second embodiment. A flowchart showing the flow of the virtual viewpoint image generation process according to the second embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The configurations shown in each embodiment are merely examples, and the present invention is not limited to the configurations shown.

Embodiment 1

In recent years, the resolution of captured images has increased due to the improvement in image quality of cameras, and the amount of data tends to increase. When transmitting the data of the multi-viewpoint image taken by a plurality of cameras to a server or the like via the network, if the data is transmitted as it is, a heavy load is applied to the network. Furthermore, the amount of calculation during the three-dimensional model generation and rendering processing on the server or the like that receives the data of the plural viewpoint image also increases. Therefore, in the present embodiment, while suppressing the network load at the time of data transmission of the multi-viewpoint image, a natural virtual viewpoint image expressed in three dimensions so that the structures and the like existing in the shooting scene become close to reality is obtained. Aspects will be described. Specifically, a mode in which a structure in a shooting scene that remains stationary or close to a stationary state is separated as an object having a unique attribute that is neither a foreground nor a background and modeled in three dimensions in advance will be described. In the following, a soccer match will be used as a shooting scene, and a case where a soccer goal as a structure is modeled in three dimensions in advance will be described as an example.

The virtual viewpoint image is an image generated by the end user and / or an appointed operator or the like freely manipulating the position and orientation of the virtual camera, and is also called a free viewpoint image or an arbitrary viewpoint image. Further, the generated virtual viewpoint image and the plurality of viewpoint images based on the virtual viewpoint image may be a moving image or a still image. In each of the embodiments described below, an example in which both the input multi-viewpoint image and the output virtual viewpoint image are moving images will be described. The structure in the present embodiment may be a static object (stationary object) whose position does not change when photographs are taken from the same angle in time series. For example, when an indoor studio or the like is used as a shooting scene, furniture and props can be treated as the structure referred to in the present embodiment.

FIG. 2 is a view showing the arrangement of all 10 camera systems 110a to 110j constituting the virtual viewpoint image generation system according to the present embodiment in a bird's-eye view of the field 200 viewed from directly above. Each camera system 110a to 110j is installed so as to surround the field 200 at a certain height from the ground, and photographs one goal in front from various angles to acquire multi-viewpoint image data having different viewpoints. .. A soccer court 201 is drawn (actually with a white line) on the grass field 200, and soccer goals are placed on the left and right ends thereof. Further, the x mark 203 in front of the soccer goal 202 on the left side indicates the common line-of-sight direction (gazing point) of the camera systems 110a to 110j, and the broken line circle 204 is photographed by the camera systems 110a to 110j centering on the gazing point 203. Shows possible areas. In the present embodiment, one corner of the field 200 is set as the origin, and the coordinate system is represented by the x-axis in the longitudinal direction, the y-axis in the lateral direction, and the z-axis in the height direction.

FIG. 3 is a diagram showing an example of the hardware configuration of the virtual viewpoint image generation system. The virtual viewpoint image generation system of FIG. 3 is composed of camera systems 110a to 110j, a switching hub 120, a control device 130, a server 140, and a database 150.

Each camera system 110a to 110j includes image pickup units 111a to 111j composed of a lens, an image pickup sensor, and the like, and camera adapters 112a to 112j that control the image pickup unit and perform predetermined image processing according to the instructions of the control device 130. ing. The camera adapter includes an arithmetic processing unit (CPU or ASIC) and memory (RAM and ROM) necessary for control and image processing. Further, between the camera systems 110a to 110j, adjacent camera systems are connected to each other by a network cable 160a to 160i in a daisy chain system. The image data captured by the camera systems 110a to 110j is transmitted via the network cables 160a to 160i. The switching hub (hereinafter referred to as “HUB”) 120 routes data transmission on the network. The HUB 120 and the camera system 110a are connected by a network cable 170a, and the HUB 120 and the camera system 110j are connected by a network cable 170b. The server 140 processes the multi-viewpoint image data transmitted from the camera systems 110a to 110j to generate virtual viewpoint image data. The server 140 also generates a time synchronization signal to control the synchronization of the entire system. The database 150 (hereinafter, referred to as "DB") accumulates image data sent from the server 140, and provides the accumulated image data to the server 150 as needed. The network cable 170c is connected between the HUB 120 and the server 140, the network cable 170d is connected between the server 140 and the DB 150, and the network cable 113e is connected between the HUB 120 and the control device 130. The control device 130 comprehensively controls each camera system 110a to 110j and the server 140. Then, the virtual viewpoint image generated by the server 140 based on the plurality of viewpoint images is output to, for example, a display device (not shown) or another information processing device on the network. In the system configuration shown in FIG. 3, a plurality of camera systems are connected by a daisy chain method, but a star type connection in which the HUB 120 and each camera system 110a to 110j are directly connected may be used. Further, the number of camera systems constituting the virtual viewpoint image generation system is not limited to 10.

Here, the acquisition of the multi-viewpoint image data in the present embodiment will be described. First, the server 140 transmits a time synchronization signal to each camera system (time server function). In the camera systems 110a to 110j, under the control of the internal camera adapters 112a to 112j, the imaging units 111a to 111j take pictures according to the received time synchronization signal. This makes it possible to acquire a multi-viewpoint image with a moving image synchronized in frame units. Specifically, the image data taken by each camera system is sequentially transmitted to the server 140 as follows. First, in the camera system 110a, the image data captured by the imaging unit 111a is subjected to image processing described later by the camera adapter 112a, and then transmitted to the camera system 110b via the network cable 160a. The camera system 110b performs the same processing, and transmits the captured image data to the camera system 110c together with the captured image data acquired from the camera system 110a. The same processing is executed in each camera system, and the captured image data for a total of 10 viewpoints acquired by each of the 10 camera systems 110a to 110j is transmitted to the HUB 120 via the network cable 170b, and finally the server. Sent to 140. The server 140 uses the received image data for 10 viewpoints to perform image processing such as generation of a structure model, estimation of the shape of an object, and rendering, which will be described later.

FIG. 4 is a diagram schematically showing the photographing regions from the imaging units 111a to 111d of the four camera systems 110a to 110d among the ten camera systems, respectively, based on FIG. 2 described above. The triangular regions 411 to 414 extending from each of the camera systems 110a to 110d represent the imaging regions corresponding to the camera systems 110a to 110d in terms of visual volume. The polygonal region 415 on which the photographing regions 411 to 414 represented by the four triangles overlap represents the common imaging region of the camera systems 110a to 110d. Here, the common imaging region has been described by taking the case of four camera systems as an example, but the common imaging region in all ten camera systems can be derived by the same method. As a matter of course, the common shooting area in all 10 camera systems is smaller than the above-mentioned polygonal area 415. The common imaging region of the camera group that captures the common gazing point can be obtained by calculating the overlapping region of the visual volume of each camera. Similarly, the three-dimensional model of the object existing in the common shooting area can be derived from the overlapping area of the multi-viewpoint image acquired by each camera system.

Next, a method of three-dimensionally modeling the structure existing in the common imaging region obtained as described above, which is one of the features of the present embodiment, will be described. Here, a case where a three-dimensional model of the soccer goal 202 is generated will be described as an example. First, volume data (see FIG. 5) in which the three-dimensional space on the field 200 is filled with cubes (voxels) having a constant size is prepared. The values of voxels constituting the volume data are represented by 0 and 1, where "1" indicates a shape region and "0" indicates a non-shape region, respectively. In FIG. 5, reference numeral 501 indicates a voxel (larger than the actual size for convenience of explanation). Next, the three-dimensional coordinates of the voxels are converted from the world coordinate system to the camera coordinate system by using the camera parameters of the imaging units 111a to 111j included in each camera system 110a to 110j. Then, when the structure is in the camera coordinate system, a model (structure model) representing the three-dimensional shape of the structure is generated by the voxel. The camera parameters refer to information such as the installation position and orientation (line-of-sight direction) of each imaging unit 111a to 111j, and the focal length of the lens.

FIG. 6 is a sequence diagram showing the generation process of the structure model existing in the shooting scene. A series of processes shown in this sequence diagram are performed before the start of shooting the main part of the multi-viewpoint image, which is the original data of the virtual viewpoint image (for example, before the start of the game), such as when setting up a stadium. In FIG. 6, a set of 10 camera systems 110a to 110j is represented as a “camera system group”.

In step 601 each of the imaging units 111a to 111j photographs the target three-dimensional space (here, the field 200) in a state where there is no structure (here, the soccer goal 202 is not installed). FIG. 7A shows an image obtained by photographing the field 200 without the soccer goal 202 from the imaging unit 111i of the camera system 110i. Such captured images with different viewpoints are acquired in each camera system.

Next, in step 602, the imaging units 111a to 111j photograph the target three-dimensional space (field 200) in a state where there is a structure (here, the soccer goal 2020 is installed). FIG. 7B shows an image obtained by photographing the field 200 with the soccer goal 202 from the imaging unit 111i of the camera system 110i. Similar to step 601 such captured images with different viewpoints are acquired in each camera system. It is assumed that the captured image data obtained in steps 601 and 602 are held in the memories in the camera adapters 112a to 112j.

In step 603, each camera adapter 112a to 112j uses the difference between the captured image obtained in step 601 and the captured image obtained in step 602 to show an image area in which the structure is shown and an image in which the other background is shown. Separate into areas. As a result, image data corresponding to the structure (here, soccer goal 202) and image data corresponding to the other background (here, field 200) can be obtained.

In step 604, each camera adapter 112a to 112j transmits the image data corresponding to the structure obtained in step 603 and the image data corresponding to the background to the server 140.

In step 605, the server 140 is three-dimensional of the structure composed of the above-mentioned voxels (here, soccer goal 202) based on the image data of the structure received from each camera system and the camera parameters of each camera system. Generate a model. FIG. 8 is a diagram showing a three-dimensional model of the soccer goal 202 on the above-mentioned volume data. The three-dimensional shape may be expressed not by the voxel itself but by a set (point cloud) of points indicating the center of the voxel. The structure model generated in this way is stored in the memory in the server 140 or in the DB 150. In addition, the background image data received together with the image data of the structure is also saved.

The above is the flow of processing when generating the structure model in the shooting scene. A three-dimensional model of another structure, such as a corner flag, may be generated in a similar manner. In the present embodiment, the structure and the background other than the structure are separated on the camera adapter side, but this may be performed on the server 140 side.

Subsequently, using the structure model obtained as described above, the generation of a virtual viewpoint image in which the structure existing in the shooting scene is expressed without discomfort will be described. FIG. 9 is a sequence diagram showing a process of generating a virtual viewpoint image according to the present embodiment. Similar to the sequence diagram of FIG. 6, a set of 10 camera systems 110a to 110j is described as a “camera system group”.

In step 901, the control device 130 sends a shooting instruction (shooting start command) of a multi-viewpoint image, which is a source of the virtual viewpoint image, to the server 140 in accordance with the start of a soccer match or the like. In the following step 902, the server 140 transmits a time synchronization signal to each of the camera systems 110a to 110j in response to a shooting instruction from the control device 130. Then, in step 903, each camera system 110a to 110j starts shooting the target three-dimensional space (here, the three-dimensional space on the field 200). As a result, for example, in the camera system 110i, an image during a soccer match as shown in FIG. 10A can be obtained. Then, such images with different viewpoints are taken in each camera system.

In step 904, each camera adapter 112a to 112j executes a process of extracting data of a foreground (here, a player and a ball) composed of moving objects from the captured image acquired in step 903. In this extraction process, the captured image acquired in step 903 and the captured image with a structure (FIG. 7 (b)) acquired in step 602 described above are compared, and the foreground and background are separated based on the difference. It can be rephrased as processing. FIG. 10B shows an image of only the foreground extracted from the captured image (panoramic image) of FIG. 10A. In the following step 905, the camera adapters 112a to 112j transmit the extracted foreground image data to the server 140. At this time, the image area (background image data) corresponding to the field 200 and the soccer goal 202 is not transmitted to the server 140. Therefore, the amount of data transmission is suppressed by that amount.

In step 906, the control device 130 transmits a virtual viewpoint image generation instruction (generation start command) to the server 140 together with information on the virtual viewpoint and the gazing point based on the user instruction. At this time, the user who wants to create and view the virtual viewpoint image inputs the information necessary for generating the virtual viewpoint image via the GUI (not shown) included in the control device 130. Specifically, information necessary for generating a virtual viewpoint image (hereinafter, referred to as "virtual viewpoint information") such as the position of the virtual viewpoint, its movement path, and where (which object) to gaze at is predetermined. Set via UI screen.

In step 907, the server 140 generates a three-dimensional model (foreground model) of a moving object in the shooting scene by using the foreground image data received from the camera group and the above-mentioned camera parameters. Here, a three-dimensional model of the player and the ball is generated as a foreground model. FIG. 11 is a diagram showing a three-dimensional model corresponding to a certain player among the three-dimensional models of the player and the ball generated in this step on the volume data as in FIG. 8 described above.

In step 908, the server 140 generates a virtual viewpoint image using the virtual viewpoint information received from the control device 130, the foreground model acquired in step 907, and the structure model and background data generated / acquired in advance. To do. Specifically, for example, using the Visual Hull method or the like, the shapes of the structure model and the foreground model when viewed from the set virtual viewpoint (virtual camera) are estimated. As a result of this shape estimation process, volume data expressing the three-dimensional shape of the object existing in the shooting scene can be obtained. Once the three-dimensional shapes of the objects viewed from the virtual viewpoint are obtained in this way, the three-dimensional shapes of these objects are then combined into one image. In the synthesis process, if the distance from the set virtual viewpoint is closer to the foreground model than to the structure model, the foreground model is mapped from above the structure model. On the contrary, when the structure model is closer to the virtual viewpoint than the foreground model, the structure model is mapped from above the foreground model. In this way, for example, when the point at which the viewpoint from the imaging unit 111i of the camera system 110i is moved in the height direction (+ z direction) is used as the virtual viewpoint, the virtual viewpoint image is the image as shown in FIG. 10 (c). Become. In the virtual viewpoint image shown in FIG. 10C, it can be seen that the player and the ball, which are the foreground model, and the soccer goal, which is the structure model, are all mapped on the field 200 in a natural three-dimensional shape. .. By repeating such processing for the number of time frames set separately, a desired virtual viewpoint image by moving image can be obtained.

In the present embodiment, in the sequence of FIG. 9, the total amount of data transmission is suppressed by not transmitting any background image data. In this case, for example, when shooting an outdoor sports scene as a moving image, the sunshine conditions and the like change in time series, so that there may be a problem that the background portion in the completed virtual viewpoint image is different from the actual one. When such a problem is a concern, the background image data obtained in the foreground / background separation in step 904 may be appropriately transmitted between the transmission of the foreground image data.

Further, in the present embodiment, the structure model and the foreground model are generated by the server 140, but the present invention is not limited to this. For example, the structure model may be generated by the camera adapter and transmitted to the server 140. Alternatively, the server 140 may acquire the data of the structure model generated by another information processing device. The point is that the structure model may be available on the server 140 at the stage of generating the foreground model from the foreground data extracted from the plurality of viewpoint images.

<Modification example>
In the above example, the structure in the shooting scene is treated as an object having a unique attribute that is neither the foreground nor the background, and a three-dimensional model of the structure is generated and held in advance to reduce the amount of data transmission. From the viewpoint of reducing the amount of data transmission, the purpose can be achieved by treating the three-dimensional model of the structure as a background. However, when the structure model is used as the background, the following problems occur.

FIG. 12 is a diagram showing the arrangement of all 10 camera systems 110a to 110j constituting the virtual viewpoint image generation system according to the present modification when the shooting scene is sumo. Each of the camera systems 110a to 110j is installed on the ceiling of the sumo venue so as to surround the ring, and photographs the top of the ring from various angles to acquire multi-viewpoint image data having different viewpoints. In this case, a three-dimensional model is performed based on an image of only the ring (= structure), and the obtained three-dimensional shape of the ring is treated as a background.

Here, it is assumed that as a result of two wrestlers wrestling, one wrestler has fallen from the ring as shown in FIG. 13 (a), for example. Consider a case in which the state of FIG. 13A is photographed by all 10 camera systems 110a to 110j and only the foreground image data is transmitted to the server 140. The server 140, which receives the image data of the foreground on the server 140, maps the two wrestlers who are the foreground on the three-dimensional model of the ring prepared in advance as the background. Then, as shown in FIG. 13B, the image looks as if a wrestler who should have been pushed out and fell outside the ring is lying on the ring. That is, when a three-dimensional modeled structure is treated as a background, it does not become a natural virtual viewpoint image depending on the position of the foreground. Therefore, when treating the structure model as a background, it is determined in advance whether or not a natural virtual viewpoint image can be obtained, and a warning is given to the user when there is a high possibility that the virtual viewpoint image will be unnatural. Is desirable.

FIG. 14 is a bird's-eye view of the ring from directly above, and the area around the ring is divided into four areas, A, B, C, and D. Each of the areas A, B, C, and D shows the part under the ring (outside the ring). The x mark in the center is the gazing point of the imaging units 111a to 111j in the camera systems 110a to 110j. In this modified example, when a virtual viewpoint image generation instruction is given, the position of the foreground is confirmed. In the above example, whether or not the wrestler's position is on the ring is determined based on the distance from the specified virtual viewpoint (virtual camera) or the image of the camera that is taking a bird's-eye view of the entire ring (not shown). To do. Then, if at least one of the wrestlers is not on the ring and the position of the designated virtual viewpoint and the position of the wrestler do not exist in the same area of any of A to D, the virtual viewpoint image cannot be generated. And give a warning. If the area where the virtual viewpoint position exists and the area where the wrestler is located are different, such as one in the area of A and the other in the area of C, the wrestler is pasted in a place different from the actual one. This is because there is a high possibility that such an unnatural virtual viewpoint image will be generated. In this way, care must be taken when treating the structure model as a background.

According to this embodiment, a three-dimensional model of the structure is created in advance and treated differently from other foreground models. This makes it possible to generate a virtual viewpoint image in which the structure in the shooting scene is expressed without discomfort while suppressing the amount of data transmission of the plurality of viewpoint images that are the basis of the virtual viewpoint image.

Embodiment 2

The first embodiment is an embodiment in which the amount of data transfer is suppressed by separating the structure in the shooting scene as an object having a unique attribute that is neither the foreground nor the background, modeling it in advance in three dimensions, and holding it in the server. .. Next, a mode in which the amount of data transfer is suppressed by thinning out and transmitting data for the structure while treating the structure in the shooting scene as the foreground will be described as the second embodiment. The contents common to the first embodiment such as the system configuration will be omitted or simplified, and the differences will be mainly described below.

Also in this embodiment, the same as in the first embodiment, the case where a soccer match is used as a shooting scene will be described as an example. That is, the following description will be given on the assumption that the arrangement of the camera system is the same as that in FIG. In this case, the soccer goal, which is a structure, is treated as a foreground model while being distinguished from the players and the ball. FIG. 15 is a flowchart showing a flow of processing for thinning out and transmitting image data of a structure portion in a shooting scene according to the present embodiment. The flow of FIG. 15 starts execution in each camera system when a user instructs the user to take a multi-viewpoint image which is a source of the virtual viewpoint image via the UI of the control device 130. That is, it is realized by the CPU or the like in the camera adapter executing a predetermined program.

Here, it is necessary to complete the preparatory process before starting the execution of the flow of FIG. Specifically, in each camera system 110a to 110j, panoramic images (see FIGS. 7A and 7B) taken in the field 200 with and without a structure are acquired, and each camera It is stored in the memory in the adapters 112a to 112j. This preparatory process is performed, for example, at the time of setting up the stadium before the start of the game. The data of these images obtained in the preparatory process is also transmitted to the server 140, and is held in the memory in the server 140 for reference in the virtual viewpoint image generation process described later. The flow of FIG. 15 can be executed on the premise that such a preparatory process is completed.

First, in step 1501, the values of the counters (not shown) held inside the camera adapters 112a to 112j are initialized. Specifically, "0" is set as the initial value. In the following step 1502, the imaging units 111a to 111j start taking pictures according to the time synchronization signal transmitted from the server 140. Next, in step 1503, the subsequent processing is separated depending on whether or not the current counter value is “0”. If the counter value is "0", the process proceeds to step 1507, and if the counter value is other than "0", the process proceeds to step 1504.

In step 1504, the counter value is decremented by "1". In the following step 1505, each camera adapter 112a to 112j executes a process of extracting a foreground region from the images (frames) captured by the imaging units 111a to 111j. Specifically, the process of obtaining the difference from the captured image (foreground / background separation process) is performed using the panoramic image with a structure out of the two patterns of panoramic images acquired and held in the preparatory process. Of the two patterns of panoramic images acquired in the preparatory process, the panoramic image with a structure shows the soccer goal 202 as a structure installed on the field 200 (FIG. 7 (b)). .. Therefore, an image obtained by cutting out an area in which only dynamic objects such as players and balls are shown, which does not include a soccer goal, can be obtained as foreground data. Then, in step 1506, each camera adapter 112a to 112j transmits the foreground data not including the structure obtained in step 1505 to the server 140. When the transmission of the foreground data is completed, the process proceeds to step 1510, and it is determined whether or not the shooting is completed. If the instruction to end shooting has not been received from the server 140, the process returns to step 1503.

In step 1507, each camera adapter 112a to 112j executes a process of extracting a foreground region from images (frames) captured by the imaging units 111a to 111j. Specifically, the foreground / background separation process for obtaining the difference from the captured image is performed using the panoramic image without a structure out of the two patterns of panoramic images acquired / held in the preparatory process. Of the two patterns of panoramic images acquired in the preparatory process, only the field 200 in which the soccer goal 202 is not installed is shown in the panoramic image without structures (FIG. 7A). Therefore, an image obtained by cutting out not only the area where the player or the ball is shown but also the area where the soccer goal is shown can be obtained as the foreground data. That is, in this step, the soccer goal, which is a structure, is also extracted as the foreground. Then, in step 1508, each camera adapter 112a to 112j transmits the "foreground data including the structure" obtained in step 1507 to the server 140. At this time, information indicating the presence or absence of the structure (for example, "1" when it is included and "0" when it is not included is set so that the server 140 that received the data can know that the area of the structure is also included in the foreground data. The indicated binary flag) is added and transmission is performed. In the following step 1509, a predetermined value N (N> 1) is set in the counter. Specifically, when the frame rate of moving image shooting by each imaging unit 111a to 111j is 60 fps, a value such as "60" is set. The user can freely change the frequency (once every N times) of transmitting the foreground data including the structure by setting a predetermined value set in the counter to an arbitrary value. After completing the setting of the predetermined value for the counter, the process proceeds to step 1510, and it is determined whether or not the shooting is completed. If the instruction to end shooting has not been received from the server 140, the process returns to step 1503.

The above is the content of the process of thinning out and transmitting the image data of the structure portion in the shooting scene. As a result of performing such processing, for example, when the same value as the frame rate is set in the counter as a predetermined value, the transmission of the foreground image data including the structure (here, the soccer goal) is performed only once in 60 times by the server. It will be done against 140. Of course, dynamic objects such as players and balls are transmitted all 60 times (every frame). In this way, the image information of the structure, which is a static object, can be transmitted at a lower frame rate than the dynamic object such as a player or a ball, so that the image data of the foreground including the structure is transmitted every frame. Transmission efficiency can be significantly improved compared to the above. Further, the transmission data can be reduced by transmitting the foreground image including the image information of the structure at a lower frequency than the foreground image not including the image information of the structure.

Next, a process for generating a virtual viewpoint image on the server 140 based on the foreground image data sent sequentially as described above will be described. FIG. 16 is a flowchart showing the flow of the virtual viewpoint image generation process on the server 140. The flow of FIG. 16 targets foreground images for a specific time frame (for example, 10 seconds) specified by the user from all the foreground image data captured and transmitted by the camera systems 110a to 110j. It is executed on a frame-by-frame basis. Further, this series of processes is realized by the CPU in the server 140 executing a predetermined program based on the instruction of the control device 130.

First, in step 1601, the foreground image (frame) of interest to be processed is determined from the foreground image data for the set time frame. In the following step 1602, whether or not the foreground image of interest contains a structure is determined based on the above-mentioned binary flag. As a result of the determination, if the foreground image of interest contains a structure, the process proceeds to step 1603, and if the foreground image of interest does not include a structure, the process proceeds to step 1605.

In step 1603 when the structure is included in the foreground image of interest, an image area corresponding to the structure is extracted from the foreground image of interest, and an image representing the structure (hereinafter referred to as “structure image”) is generated. .. This generation process is performed by the following procedure. First, the difference between the captured image (overall view image) with the structure acquired in the above-mentioned preparatory process and the foreground image of interest is obtained, and the image area corresponding to the foreground is extracted. Next, the image area corresponding to the extracted foreground and the captured image (entire view image) in a state without a structure held in advance are combined. Then, the difference between the composite image obtained by the synthesis and the foreground image of interest is obtained, and a structure image representing only the image region corresponding to the structure is obtained. Then, in step 1604, the data of the structure image generated in step 1603 is held in the memory in the server 140. If the structure image data is already retained, it will be overwritten (updated) by the newly generated structure image data. After saving the data of the generated structure image in the memory, the process proceeds to step 1607.

On the other hand, in step 1605 when the structure is not included in the foreground image of interest, the data of the structure image generated and held by the preceding processes of steps 1603 and 1604 is read out. In the following step 1606, the read structure image and the attention foreground image not including the structure are combined to generate the attention foreground image including the structure.

In step 1607, a three-dimensional model (foreground model) of an object in the shooting scene is generated with the structure as a part of the foreground. At this time, in this step when the structure is originally included in the attention foreground image (Yes in step 1602), the foreground model is generated using the attention foreground image as it is. On the other hand, in this step when the structure is not originally included in the foreground image of interest (No in step 1602), a foreground model is generated using the foreground image of interest in which the structure is synthesized in step 1606. In either case, a foreground model that includes a soccer goal, which is a structure (static object), is generated in addition to dynamic objects such as players and balls.

In step 1608, the shape of the foreground model generated in step 1607 when viewed from the virtual viewpoint is estimated based on the position information of the virtual viewpoint separately set by the user, and the virtual viewpoint image is generated.

The above is the content of the virtual viewpoint image generation process on the server 140 according to the present embodiment. As in the present embodiment, the same effect as that of the first embodiment can be obtained by treating the structure in the shooting scene as the foreground and thinning out the image data for transmission.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

110a-110j Camera system 112a-112j Camera adapter 140 server

Claims (26)

The first generation means for generating 3D shape data corresponding to objects photographed from a plurality of directions, and
A first acquisition means for acquiring 3D shape data corresponding to a structure photographed from a plurality of directions,
A second acquisition means for acquiring background data corresponding to at least a background different from the object and the structure, which is photographed from a plurality of directions.
A third acquisition means for acquiring information indicating the specified viewpoint,
The three-dimensional shape data corresponding to the object generated by the first generation means, the three-dimensional shape data corresponding to the structure acquired by the first acquisition means, and the three-dimensional shape data acquired by the second acquisition means. A second generation means that generates an image based on the background data and information indicating the viewpoint acquired by the third acquisition means.
Have,
The first generation means is based on an image including the area of the object, the area of the structure and the background area, and an image not including the area of the object but including the area of the structure and the background area. The system is characterized in that it can generate three-dimensional shape data corresponding to the object. The system according to claim 1, wherein the first acquisition means acquires three-dimensional shape data corresponding to the structure generated based on a photographed image obtained by photographing before the start of an event. The first acquisition means is a plurality of images based on a plurality of captured images acquired by shooting from a plurality of shooting directions before the start of the event, and represents a region of the structure in a distinctive manner from other regions. The system according to claim 1 or 2, wherein the three-dimensional shape data corresponding to the structure generated based on the image of the above is acquired. Any of claims 1 to 3, wherein the first acquisition means generates and acquires three-dimensional shape data corresponding to the structure based on a photographed image obtained by photographing before the start of the event. The system according to item 1. The first acquisition means is a plurality of images based on a plurality of captured images acquired by shooting from a plurality of shooting directions before the start of the event, and represents a region of the structure in a distinctive manner from other regions. The system according to any one of claims 1 to 4, wherein the three-dimensional shape data corresponding to the structure is generated and acquired based on the image of the above. The system according to any one of claims 1 to 5, wherein the first acquisition means acquires three-dimensional shape data corresponding to the structure before the start of an event. The system according to claim 2, wherein the captured image acquired by photographing before the start of the event is an image that does not include the region of the object but includes the region of the structure and the background region. The first generation means is capable of generating three-dimensional shape data corresponding to the object based on a plurality of captured images acquired by photographing from a plurality of photographing directions after the start of the event. Item 6. The system according to any one of Items 1 to 7. The first generation means is based on a plurality of captured images acquired by shooting from a plurality of shooting directions after the start of the event and a plurality of captured images acquired by shooting from a plurality of shooting directions before the start of the event. The system according to any one of claims 1 to 8, wherein the three-dimensional shape data corresponding to the object can be generated. 8. The photographed image acquired by photographing from a plurality of photographing directions after the start of the event is an image including the area of the object, the area of the structure, and the area of the background. Or the system according to 9. The third-dimensional shape data of the structure, the background data, and the information indicating the viewpoint are according to any one of claims 1 to 3, wherein the information indicating the viewpoint is acquired by a communication means included in the system. system. A system used to generate images based on a specified viewpoint.
A first generation means for generating an image including a region of a structure based on a captured image,
A second generation means for generating an image that includes an area of an object and does not include an area of the structure based on a captured image.
A transmission means for transmitting an image generated by the first generation means and an image generated by the second generation means, and
Based on the image generated by the first generation means and the image generated by the second generation means transmitted by the transmission means, an image representing the region of the structure to be distinguished from other regions is generated. 3 generation means and
Have,
The transmission means can transmit the image generated by the first generation means at a lower frequency than the image generated by the second generation means.
A system characterized by that. A system used to generate images based on a specified viewpoint.
A first generation means for generating an image including a region of a structure based on a captured image,
A second generation means for generating an image that includes an area of an object and does not include an area of the structure based on a captured image.
A transmission means for transmitting an image generated by the first generation means and an image generated by the second generation means, and
Based on the image generated by the first generation means and the image generated by the second generation means transmitted by the transmission means, the three-dimensional shape data corresponding to the object and the three dimensions corresponding to the structure are obtained. A third generation means for acquiring shape data and
Have,
The transmission means can transmit the image generated by the first generation means at a lower frequency than the image generated by the second generation means.
A system characterized by that. It further has a fourth generation means that generates an image based on the designated viewpoint based on the image generated by the first generation means and the image generated by the second generation means transmitted by the transmission means. The system according to claim 12 or 13 . The system according to any one of claims 1 to 14 , wherein the object is a moving object. The system according to any one of claims 1 to 15 , wherein at least one of the person and the ball is the object. The system according to any one of claims 1 to 16 , wherein the structure is an object that remains stationary. The system according to any one of claims 1 to 16 , wherein at least one of a soccer goal and a corner flag used in a soccer match is the structure. The system according to any one of claims 1 to 18 , wherein the structure is an object installed at a predetermined position. The system according to any one of claims 1 to 19 , wherein at least a part of the structure is installed on a field where a person who is an object competes. The system according to any one of claims 1 to 20 , wherein the structure is a designated object. A generation method that generates an image based on a specified viewpoint.
The first generation step of generating 3D shape data corresponding to objects photographed from multiple directions, and
The first acquisition process for acquiring 3D shape data corresponding to structures photographed from multiple directions, and
A second acquisition step of acquiring background data corresponding to at least a background different from the object and the structure, which is photographed from a plurality of directions.
The third acquisition process to acquire information indicating the specified viewpoint, and
The three-dimensional shape data corresponding to the object generated by the first generation step, the three-dimensional shape data corresponding to the structure acquired by the first acquisition step, and the three-dimensional shape data corresponding to the structure acquired by the second acquisition step. A second generation step of generating an image based on the background data and the information indicating the viewpoint acquired by the third acquisition step.
Have,
In the first generation step, based on an image including the area of the object, the area of the structure and the background area, and an image not including the area of the object and including the area of the structure and the background area. A generation method characterized in that three-dimensional shape data corresponding to the object is generated. A plurality of images based on an image acquired by shooting from a plurality of shooting directions before the start of the event in the first acquisition step, which does not include the area of the object but includes the area of the structure and the background area. 22. The generation according to claim 22 , wherein three-dimensional shape data corresponding to the structure is generated and acquired based on a plurality of images representing the region of the structure in a distinctive manner from other regions. Method. A generation method that generates an image based on a specified viewpoint.
The first generation step of generating an image including a region of a structure based on a captured image, and
The second generation step of generating an image including the area of the object and not including the area of the structure based on the captured image, and
A transmission step of transmitting an image generated by the first generation step and an image generated by the second generation step.
Based on the image generated in the first generation step and the image generated in the second generation step transmitted in the transmission step, an image representing the region of the structure to be distinguished from other regions is generated. 3 generation process and
Have,
A generation method characterized in that, in the transmission step, the image generated by the first generation step is transmitted at a lower frequency than the image generated by the second generation step. A generation method that generates an image based on a specified viewpoint.
The first generation step of generating an image including a region of a structure based on a captured image, and
The second generation step of generating an image including the area of the object and not including the area of the structure based on the captured image, and
A transmission step of transmitting an image generated by the first generation step and an image generated by the second generation step.
Based on the image generated in the first generation step and the image generated in the second generation step transmitted in the transmission step, the three-dimensional shape data corresponding to the object and the three-dimensional corresponding to the structure are obtained. The third generation process to acquire shape data and
Have,
In the transmission step, the image generated by the first generation step is transmitted at a lower frequency than the image generated by the second generation step.
A generation method characterized by that. A program for operating a computer as each means of the system according to any one of claims 1 to 21 .

Images