JP7080586B2 - Information processing equipment, information processing methods and programs - Google Patents

Description

The present invention relates to a technique for generating a virtual viewpoint image.

Conventionally, a technique has been proposed in which a subject is photographed from a plurality of directions by a plurality of cameras installed at different positions, and a virtual viewpoint image is generated using a plurality of captured images (multi-viewpoint images) obtained by the shooting. There is. The virtual viewpoint image generated in this way is an image representing the appearance in a virtual viewpoint not limited to the viewpoint of the installed camera. According to the technology that generates virtual viewpoint images, for example, it is possible to view a shot soccer or basketball game from various viewpoints that can be freely specified, and to view a higher sense of reality than conventional shot images. Can be given to a person.

Patent Document 1 describes that an advertisement, which is a virtual object not included in a multi-viewpoint image, is inserted into the virtual viewpoint image and displayed. Further, it is described that a planar area in real space such as a wall included in the multi-viewpoint image is selected as the position for inserting the advertisement.

Japanese Unexamined Patent Publication No. 2012-48639

However, in the technique described in Patent Document 1, when a virtual object is inserted into the virtual viewpoint image, at least a part of a subject such as a wall included in the shooting range by the camera is shielded by the virtual object in the virtual viewpoint image. It ends up.

The present invention has been made in view of the above problems, and when a virtual object is inserted into a virtual viewpoint image, there is a risk that the subject included in the shooting range of the camera will be shielded by the virtual object in the virtual viewpoint image. The purpose is to reduce.

In order to solve the above problems, the information processing apparatus according to the present invention has, for example, the following configuration. That is, a data acquisition means for acquiring image data corresponding to an image taken based on shooting by a plurality of shooting devices that shoot a shooting range from different positions, and a viewpoint showing the image data acquired by the data acquisition means and a virtual viewpoint. The first is to determine a specific area corresponding to the outside of the shooting range by the plurality of photographing devices from a plurality of candidate areas defined as areas that do not block a predetermined subject included in the virtual viewpoint image generated based on the information. It has a determination means and a second determination means for determining the position of the virtual object so that the virtual object that does not exist in the specific area is displayed at the position corresponding to the specific area in the virtual viewpoint image. ..

According to the present invention, when a virtual object is inserted into a virtual viewpoint image, it is possible to reduce the possibility that the subject included in the shooting range of the camera is obscured by the virtual object in the virtual viewpoint image.

It is a figure for demonstrating the structure of the image processing system 10. It is a figure for demonstrating the functional structure of the information processing apparatus 100. It is a flowchart for demonstrating the process which concerns on the generation of the virtual viewpoint image by the information processing apparatus 100. It is a flowchart for demonstrating the process which concerns on the identification of the non-photographing area by the information processing apparatus 100. It is a figure for demonstrating the non-photographing area using a bird's-eye view. It is a figure for demonstrating the non-photographing area using the sectional view. It is a figure for demonstrating the non-photographing area corresponding to a virtual viewpoint. It is a flowchart for demonstrating the process which concerns on the arrangement of the virtual advertisement by the information processing apparatus 100. It is a flowchart for demonstrating the process which concerns on the determination of the placement candidate of the virtual advertisement by the information processing apparatus 100. It is a figure for demonstrating the placement candidate of a virtual advertisement. It is a flowchart for demonstrating the process which concerns on the determination of the direction of a virtual advertisement by an information processing apparatus 100. It is a figure for demonstrating the example of the arrangement of the virtual advertisement according to the virtual viewpoint. It is a figure for demonstrating the example of the arrangement of the virtual advertisement according to the virtual viewpoint. It is a figure for demonstrating the virtual viewpoint image in which a virtual advertisement is inserted.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments.

[System configuration]
FIG. 1 is a diagram showing a configuration example of the image processing system 10 according to the present embodiment. The image processing system 10 includes an information processing device 100, a plurality of photographing devices (cameras) 108, and a terminal device 109. The plurality of photographing devices 108 perform photographing in synchronization from different directions, and input data based on the photographing into the information processing apparatus 100. In the present embodiment, the case where the data based on the shooting input from the plurality of shooting devices 108 to the information processing device 100 is a plurality of shot images (multiple viewpoint images) will be mainly described, but the present invention is not limited to this. For example, data generated based on the difference between a plurality of captured images may be input as data based on imaging.

The information processing device 100 generates and outputs a virtual viewpoint image based on the data acquired from the plurality of photographing devices 108. The terminal device 109 acquires a virtual viewpoint image output from the information processing device 100 and displays it on a display unit such as a display. The terminal device 109 is, for example, a device owned by a broadcasting station or a viewer, and specifically, a PC, a smartphone, or the like.

The virtual viewpoint image in the present embodiment represents an image obtained when a subject is photographed from a virtual viewpoint (virtual viewpoint). In other words, a virtual viewpoint image is an image that represents the appearance at a specified viewpoint. The virtual viewpoint may be specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (virtual viewpoint image) corresponding to the viewpoint arbitrarily designated by the user. Further, an image corresponding to a viewpoint designated by the user from a plurality of candidates and an image corresponding to the viewpoint automatically designated by the device are also included in the virtual viewpoint image. In the present embodiment, unless otherwise specified, the word "image" will be described as including the concepts of both moving images and still images. That is, the image processing system 10 of the present embodiment can process both still images and moving images.

In the present embodiment, a virtual object that is not included in the plurality of viewpoint images taken by the photographing apparatus 108 may be inserted into the virtual viewpoint image output from the information processing apparatus 100. The content expressed by the wording of inserting a virtual object into the virtual viewpoint image is not limited to the form in which the virtual viewpoint image including the virtual object is generated by the rendering process. For example, a form in which a virtual viewpoint image and a virtual object image are generated by independent rendering processes and the generated virtual viewpoint image and the virtual object image are displayed so as to overlap each other is also included. Further, the insertion direction is not limited, and for example, a form in which a virtual viewpoint image is later combined with the generated virtual object image is also included.

In this embodiment, the case where the virtual object inserted in the virtual viewpoint image is a virtual advertisement for displaying an advertisement in the virtual viewpoint image will be mainly described. However, virtual objects are not limited to advertisements. For example, as a virtual object, information on the progress of the game or statistical information on the game may be displayed, or characters or images explaining a subject such as a person, biological information of the person, or the like may be displayed. Further, in the present embodiment, the case where the virtual object is a planar object will be mainly described, but it may be a three-dimensional object such as a rectangular parallelepiped or a sphere.

Next, the hardware configuration of the information processing apparatus 100 will be described. The hardware configuration of the terminal device 109 is the same as that of the information processing device 100. The information processing device 100 includes a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, an operation unit 105, a communication I / F (interface) 106, and a system bus 107.

The CPU 101 uses computer programs and data stored in the ROM 102 and the RAM 103 to perform calculations and logical judgments for various processes, and controls each component connected to the system bus 107. The information processing apparatus 100 may have one or more dedicated hardware or GPU (Graphics Processing Unit) different from the CPU 101, and the GPU or dedicated hardware may perform at least a part of the processing by the CPU 101. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 102 is a program memory, and stores programs and parameters for the CPU 101 to execute various processes described later. The RAM 103 is a data memory, and temporarily stores programs and data supplied from the auxiliary storage device 104, data supplied from the outside via the communication I / F 106, and the like. Further, the RAM 103 functions as a work area when the CPU 101 executes a program, a data save area when performing error processing, an area for loading a program, and the like.

The auxiliary storage device 104 is composed of, for example, a hard disk drive or the like, and stores content data such as still images and moving images, programs, and the like. The auxiliary storage device 104 is not limited to the hard disk drive, and may be an external storage device that plays a similar role. Here, the external storage device is realized by, for example, a medium (recording medium) and an external storage drive for realizing access to the media. As such media, for example, FD (flexible disk), CD-ROM, DVD, USB memory, MO (magneto-optical disk), flash memory and the like are used. Further, the auxiliary storage device 104 may be a server device or the like connected to the information processing device 100 via a network.

The display unit 110 is composed of, for example, a liquid crystal display or the like, and displays a GUI (Graphic User Interface) for the user to operate the information processing apparatus 100. The operation unit 105 is composed of input devices such as a joystick, a game pad, a touch panel, a keyboard, and a mouse, and inputs various instructions to the CPU 101 in response to an operation by the user. The communication I / F 106 communicates with an external device such as a terminal device 109 or a photographing device 108. For example, when the information processing device 100 is connected to an external device by wire, a LAN cable or the like is connected to the communication I / F 106. When the information processing device 100 has a function of wirelessly communicating with an external device, the communication I / F 106 includes an antenna. The system bus 107 connects each part of the information processing apparatus 100 to transmit information.

In the present embodiment, the display unit 110 and the operation unit 105 exist inside the information processing device 100, but the information processing device 100 does not have to include at least one of the display unit 110 and the operation unit 105. Further, at least one of the display unit 110 and the operation unit 105 exists as another device outside the information processing device 100, and the CPU 101 controls the display control unit 110 and the operation unit 105. It may operate as a control unit.

[Functional configuration]
FIG. 2 is a block diagram showing a functional configuration example of the information processing apparatus 100 according to the present embodiment. The information processing apparatus 100 includes an image acquisition unit 201, a separation processing unit 202, a foreground model generation unit 203, a foreground texture determination unit 204, a background model acquisition unit 205, a background texture determination unit 206, a camera parameter acquisition unit 207, and an area identification unit. Has 208. The information processing apparatus 100 also includes a gaze point acquisition unit 209, a candidate determination unit 210, a viewpoint acquisition unit 211, an arrangement determination unit 212, an advertisement model generation unit 213, an advertisement texture determination unit 214, a rendering processing unit 215, and an output unit 216. Have. Each of these functional units included in the information processing apparatus 100 is realized by the CPU 101 expanding the program stored in the ROM 102 into the RAM 103 and executing processing according to each flowchart described later. It should be noted that at least a part of the functional unit shown in FIG. 2 may be realized by one or a plurality of dedicated hardware or GPU different from the CPU 101.

The image acquisition unit 201 acquires data based on imaging from different directions by the plurality of imaging devices 108. Here, it is assumed that the image acquisition unit 201 acquires a plurality of captured images (multiple viewpoint images). However, the present invention is not limited to this example, and for example, data generated based on the difference between a plurality of captured images may be acquired by the image acquisition unit 201.

The separation processing unit 202 separates the captured image (multi-viewpoint image) acquired by the image acquisition unit 201 into a foreground corresponding to a predetermined subject and a background corresponding to a subject different from the predetermined subject. Here, the predetermined subject is, for example, a subject that matches a predetermined shape pattern, a moving subject, or the like. For example, when the target photographed by the photographing device 108 is a soccer match in a stadium, the player or the ball is processed as the foreground. On the other hand, a subject different from a predetermined subject to be processed as a foreground, such as a stadium field, spectator seats, or a roof, is processed as a background. The subject to be photographed by the photographing apparatus 108 is not limited to a soccer match, but may be another sport such as rugby or sumo. Further, the shooting target is not limited to the competition at the stadium, but may be a concert on the stage or the like.

As a method of separating the foreground and the background, for example, when a moving subject is used as the foreground, the pixel value changes significantly in the image captured by the photographing apparatus 108 with the pixel value unchanged or small in a plurality of frames as the background. There is a method to make the area the foreground. When the background subtraction method is used as another method, the background image taken in advance in the absence of a predetermined subject as the foreground is retained, and the foreground is extracted by taking the difference between the shot image and the background image. You can also do it.

The foreground model generation unit 203 generates model information regarding a three-dimensional model representing the shape of a subject (for example, a player or a ball) corresponding to the foreground, based on the foreground image separated by the separation processing unit 202 from each of the plurality of viewpoint images. do. The model information includes, for example, information for specifying a three-dimensional model of the subject and information regarding the position of the subject in the three-dimensional coordinate system. The three-dimensional coordinate system in the present embodiment is a coordinate system that defines a virtual three-dimensional space in which the information processing apparatus 100 arranges a three-dimensional model in order to generate a virtual viewpoint image, and is a position in the three-dimensional coordinate system. And the position in the real space have a correspondence relationship. As a method of determining the three-dimensional model of the subject, for example, there is a method of using a visual hull representing the three-dimensional shape of the subject calculated by using the two-dimensional silhouette of the subject in each of the plurality of viewpoint images.

The foreground texture determination unit 204 generates foreground model information including information about the texture by mapping a texture representing the color and texture of the subject to the foreground three-dimensional model determined by the foreground model generation unit 203. Specifically, the foreground texture determination unit 204 maps the image of the foreground separated by the separation processing unit 202 to a point cloud or polygon representing a three-dimensional model as a texture.

The background model acquisition unit 205 generates model information of a subject (for example, a stadium) corresponding to the background based on the background image separated by the separation processing unit 202 from the plurality of viewpoint images. The background model information may be stored in the auxiliary storage device 104 in advance, and the background model acquisition unit 205 may acquire the background model information by reading the background model information from the auxiliary storage device 104. The background texture determination unit 206 maps the background image separated by the separation processing unit 202 as a texture to the three-dimensional model of the background determined by the background model acquisition unit 205.

The camera parameter acquisition unit 207 acquires external parameters and internal parameters of each of the plurality of photographing devices 108. The external parameter is information regarding the position and orientation of the photographing device 108, and the internal parameter is information regarding the angle of view and focal length of the photographing device 108. Based on the model information of the foreground and the model information of the background, the area specifying unit 208 specifies a non-photographing area corresponding to the outside of the shooting range by the photographing device 108 among the areas to be drawn of the virtual viewpoint image. The non-photographed area specified here is used to determine the position in which the virtual advertisement is inserted in the virtual viewpoint image. Specific details of the non-photographed area will be described later.

The gaze point acquisition unit 209 specifies the gaze point according to the shooting direction of the plurality of shooting devices 108. In the present embodiment, the gazing point is a position facing the plurality of photographing devices 108, that is, a position corresponding to the center of the field of view of the plurality of photographing devices 108. In the image processing system 10, the orientation of each of the plurality of photographing devices 108 can be changed by the pan head, and for example, the position of the gazing point can be changed according to the situation of the game to be photographed. Then, the gaze point acquisition unit 209 newly specifies the gaze point according to the change in the position of the gaze point. In the image processing system 10, there may be a plurality of gazing points. That is, among the plurality of photographing devices 108, the plurality of photographing devices 108 included in the first group are oriented toward the first gazing point, and the plurality of photographing devices 108 included in the second group are directed to the first gazing point. It may be suitable for a second gaze point different from that of.

The candidate determination unit 210 determines an area that is a candidate for placement of a virtual advertisement in the three-dimensional coordinate system based on the model information of the foreground, the model information of the background, the gazing point, and the like. The viewpoint acquisition unit 211 acquires viewpoint information for specifying a virtual viewpoint related to the generation of a virtual viewpoint image. The viewpoint information is information according to the designation of the virtual viewpoint. For example, the viewpoint acquisition unit 211 specifies the virtual viewpoint based on the user operation performed on the operation unit 105 to specify the virtual viewpoint, and obtains the viewpoint information. May be generated. Further, for example, the viewpoint acquisition unit 211 may acquire the viewpoint information regarding the virtual viewpoint designated by the terminal device 109 from the terminal device 109. The placement determination unit 212 determines the placement of the virtual advertisement in the three-dimensional coordinate system according to the candidates determined by the candidate determination unit 210.

The advertisement model generation unit 213 determines a three-dimensional model of the virtual advertisement based on the arrangement of the virtual advertisement determined by the arrangement determination unit 212, and generates model information of the virtual advertisement including the arrangement information. The advertisement texture determination unit 214 maps the image data (advertisement data) representing the content of the virtual advertisement as a texture to the three-dimensional model of the virtual advertisement determined by the advertisement model generation unit 213, thereby including information about the texture. Generate model information for ads. It is assumed that the advertisement data is stored in the auxiliary storage device 104 in advance, and the advertisement texture determination unit 214 reads the advertisement data from the auxiliary storage device 104 and uses it.

The rendering processing unit 215 generates a virtual viewpoint image corresponding to the viewpoint information acquired by the viewpoint acquisition unit 211 based on the foreground model information, the background model information, and the virtual advertisement model information reflecting the texture mapping. do. In this generation process, the rendering processing unit 215 determines an insertion area for inserting the virtual advertisement in the virtual viewpoint image based on the arrangement information included in the model information of the virtual advertisement generated by the advertisement model generation unit 213. Then, the rendering processing unit 215 generates a virtual viewpoint image in which the virtual advertisement is inserted in the determined insertion area. The output unit 216 outputs the virtual viewpoint image generated by the rendering processing unit 215 to the terminal device 109, so that the display unit of the terminal device 109 displays the virtual viewpoint image.

The information processing apparatus 100 may control the insertion of a virtual object into a determined insertion region in the virtual viewpoint image, and the method is not limited to the one that generates the virtual viewpoint image in which the virtual advertisement is inserted. For example, the rendering processing unit 215 may generate a virtual viewpoint image in which a virtual advertisement is not inserted, based on the model information of the foreground and the model information of the background. Then, the output unit 216 includes a virtual viewpoint image in which the virtual advertisement is not inserted, information for specifying an insertion area determined based on the model information of the virtual advertisement, and an image of the virtual advertisement to be inserted in the insertion area. May be output to the terminal device 109. Then, the terminal device 109 synthesizes the image of the virtual advertisement in the insertion area in the virtual viewpoint image acquired from the information processing device 100 to generate the virtual viewpoint image in which the virtual advertisement is inserted, and the display unit of the terminal device 109. The virtual viewpoint image generated in may be displayed. Further, in this case, the terminal device 109 determines whether or not to synthesize the image of the virtual advertisement in the insertion area in the virtual viewpoint image based on the content of the operation on the terminal device 109 by the user, the information for identifying the user, and the like. You may judge.

Further, for example, the output unit 216 may output the virtual viewpoint image in which the virtual advertisement is not inserted and the information for specifying the insertion area to the terminal device 109. Then, a device different from the information processing device 100 may output the image of the virtual advertisement or the model information of the virtual advertisement to the terminal device 109. Then, the terminal device 109 may generate and display a virtual viewpoint image based on the information acquired from the information processing device 100 and the information acquired from a device different from the information processing device 100. There are various variations in the control for displaying the virtual viewpoint image in which the virtual object is inserted in this way.

The image generation method of the virtual viewpoint image by the information processing apparatus 100 is not limited to the one described above. For example, the information processing apparatus 100 may generate a virtual viewpoint image by transforming or synthesizing a captured image without generating a three-dimensional model.

[Operation flow]
FIG. 3 is a flowchart showing an example of the flow of processing related to the generation of the virtual viewpoint image by the information processing apparatus 100. The process shown in FIG. 3 is started at the timing when the operation unit 105 accepts the operation by the user for starting the generation of the virtual viewpoint image. The user operation for starting the generation is performed, for example, in response to the start of the game to be photographed by the photographing device 108. The generation of the virtual viewpoint image is not limited to the one performed in real time, and the generation of the virtual viewpoint image may be started after the end of the game, for example, based on the captured image recorded during the game. However, the start timing of the process shown in FIG. 3 is not limited to the above timing.

When the generated virtual viewpoint image is a moving image, the process shown in FIG. 3 is performed for each frame of the moving image. However, it is not necessary to execute all of the processing shown in FIG. 3 every frame, and a part of the processing may be omitted by reusing the information used for the processing of the previous frame. The process shown in FIG. 3 is realized by the CPU 101 of the information processing apparatus 100 expanding the program stored in the ROM 102 into the RAM 103 and executing the program. It should be noted that at least a part of the processing shown in FIG. 3 may be realized by one or more dedicated hardware or GPU different from the CPU 101.

In S301, the image acquisition unit 201 acquires a plurality of viewpoint images based on images taken from different directions by the plurality of photographing devices 108. In S302, the separation processing unit 202 performs image processing for separating each of the plurality of viewpoint images acquired by the image acquisition unit 201 into a foreground and a background. In S303, the foreground model generation unit 203 generates the foreground model information, and the background model acquisition unit 205 generates the background model information. The model information includes information for specifying a three-dimensional model representing the shape of the subject and information for specifying the position of the subject in the three-dimensional coordinate system. In S304, the foreground texture determination unit 204 determines the texture to be mapped to the three-dimensional model of the foreground, and generates the model information of the foreground including the texture information. Further, the background texture determination unit 206 determines the texture to be mapped to the three-dimensional model of the background, and generates the background model information including the texture information.

In S305, the viewpoint acquisition unit 211 acquires viewpoint information according to the designation of the virtual viewpoint related to the generation of the virtual viewpoint image. The viewpoint information includes information for specifying the position of the viewpoint and the direction of the line of sight. In S306, the area specifying unit 208 specifies a non-photographing area corresponding to the outside of the shooting range by the photographing device 108 among the areas to be drawn of the virtual viewpoint image based on the model information of the foreground and the model information of the background. .. The non-photographed area specified here is used to determine the position in which the virtual advertisement is inserted in the virtual viewpoint image. Details of the processing of S306 will be described later with reference to FIG. In S307, the placement determination unit 212 determines the placement of the advertisement, and the advertisement model generation unit 213 generates model information of the virtual advertisement. The details of the processing of S307 will be described later with reference to FIG.

In S308, the rendering processing unit 215 uses the model information of the foreground, the background, and the virtual advertisement to generate a virtual viewpoint image in which the virtual advertisement is inserted, which corresponds to the viewpoint information acquired by the viewpoint acquisition unit 211. In S309, the output unit 216 outputs the virtual viewpoint image generated by the rendering processing unit 215 to the terminal device 109, and ends the processing.

FIG. 4 is a flowchart showing the details of the process relating to the identification of the non-photographed area in S306 of FIG. At the start of the process of FIG. 4, the non-photographed area is initialized, and the non-photographed area is updated in the process of FIG. Then, the non-shooting area that is updated and finally identified becomes the target area for arranging the virtual advertisement.

In S401, the area specifying unit 208 acquires the model information of the foreground and the model information of the background. The foreground model information acquired here includes information for identifying the 3D model of the subject corresponding to the foreground, information regarding the position of the subject in the 3D coordinate system, and information regarding the texture mapped to the 3D model. include. Similarly, the background model information includes information that identifies the 3D model of the subject corresponding to the background, information about the position of the subject in the 3D coordinate system, and information about the texture mapped to the 3D model. ing.

In S402, the area specifying unit 208 determines whether or not the processing of S403 to S405 has been performed on the model information of each subject acquired in S401. If there is model information of an unprocessed subject, one of them is selected as a processing target and the process proceeds to S403. If there is no model information of the unprocessed subject, move to S406.

In S403, the area specifying unit 208 specifies an area to which the texture is not mapped in the three-dimensional model of the subject to be processed. The region to which the texture is not mapped is, that is, a portion of the polygon or point cloud representing the three-dimensional model of the subject to which the color information based on the image captured by the photographing apparatus 108 is not assigned. In S404, the area specifying unit 208 updates the non-photographed area by adding the area to which the texture specified in S403 is not mapped to the non-photographed area.

In S405, the area specifying unit 208 identifies an area in the three-dimensional coordinate system where the three-dimensional model of the subject to be processed is not located, and returns to S402. By repeating the processing of S405 by changing the subject to be processed, a region in which the 3D model of any subject is not located is specified in the 3D coordinate system. In S406, the area specifying unit 208 updates the non-photographed area by adding the area finally obtained by the process of S405, which is not located in the three-dimensional model of any subject, to the non-photographed area.

Here, the non-photographing region will be described with reference to FIGS. 5 and 6. FIG. 5A shows a bird's-eye view of the stadium to be photographed by the photographing apparatus 108. A field 502 exists inside the spectator seats 501 of the stadium. The plurality of photographing devices 108 are installed, for example, in the vicinity of the audience seats 501 or on the roof of the stadium, and photograph from above the field 502 so that the gazing point 503 is the center of the field of view. In the present embodiment, it is assumed that the gazing point 503 exists on the field surface, but the present invention is not limited to this, and the gazing point 503 may exist at a position vertically away from the field surface. Since the range 506 is included in any of the shooting ranges of the plurality of shooting devices 108, the subject corresponding to the inside of the range 506 is displayed in high image quality in the generated virtual viewpoint image. On the other hand, in the generated virtual viewpoint image, the area corresponding to the outside of the range 506 is displayed or not displayed with a lower image quality than the area corresponding to the inside of the range 506.

FIG. 5B shows a non-photographed area in the bird's-eye view of the stadium. The white area 505 is an area included in at least one of the photographing areas of the photographing apparatus 108, and the shaded area 504 is a non-imaging area not included in the photographing range of any of the photographing devices 108. In the present embodiment, the photographing apparatus 108 photographs the field 502 from diagonally above. The area 505 in FIG. 5B is a projection of the area included in the photographing range on the field surface. Therefore, in reality, the field surface immediately below the photographing device 108 is not photographed, and the field surface near the range 506 is the area to be photographed and the area where the texture mapped to the three-dimensional model of the subject can be obtained.

As described above, the background model acquisition unit 205 may acquire in advance a three-dimensional model of the entire stadium as a subject corresponding to the background. In this case, the area 504 includes an area inside the stadium where the three-dimensional model of the subject exists but no texture can be obtained, and an area outside the stadium where the three-dimensional model of the subject does not exist. 6 (a) and 6 (b) show a cross-sectional view of the stadium as viewed from the horizontal direction. FIG. 6 (b) shows the region 504 and the region 505 shown in FIG. 5 (b).

In the present embodiment, the non-shooting area specified in this way is the target area for arranging the virtual advertisement. That is, the information processing apparatus 100 determines an area in the virtual viewpoint image corresponding to the area outside the photographing range by the plurality of photographing devices 108 as an insertion area into which the virtual advertisement is inserted. As a result, the possibility that the subject included in the shooting range of the shooting device 108 is blocked by the virtual advertisement in the virtual viewpoint image is reduced. The non-photographed area includes an area in which the three-dimensional model of the subject does not exist and an area in which the three-dimensional model of the subject exists but the texture cannot be obtained. That is, the area in the virtual viewpoint image that does not correspond to the subject included in the image captured by the photographing device 108 is determined as the insertion area for inserting the virtual advertisement. In addition, a region corresponding to the inside of the subject included in the captured image and / or a region corresponding to a surface of the subject that is not included in the captured image is also determined as an insertion region. The area corresponding to the inside of the subject is, for example, the inside of a wall of a building, and the surface of the subject that is not included in the captured image is, for example, the back side of the chair of the audience seat 501.

In this embodiment, the area 504 that is not included in the shooting range of any of the shooting devices 108 is specified as the non-shooting area, and the placement of the virtual advertisement is determined based on the specified non-shooting area. Explain to. However, the present invention is not limited to this, and for example, the outside of the range 506 included in the photographing range of any photographing apparatus 108 may be specified as a non-photographing area. Further, the outside of the range included in the shooting range of the predetermined number or more of the shooting devices 108 among the plurality of installed shooting devices 108 may be specified as the non-shooting area.

In S407, the area specifying unit 208 acquires the viewpoint information held by the viewpoint acquisition unit 211. In S408, the area specifying unit 208 updates the non-photographing area obtained by the processing of S401 to S406 according to the virtual viewpoint specified from the viewpoint information, and ends the processing. In the present embodiment, the case where the non-photographing area specified by the area specifying unit 208 changes according to the virtual viewpoint will be mainly described, but the present invention is not limited to this. When the non-photographing area is determined regardless of the virtual viewpoint, the processing of S407 and S408 in FIG. 4 is omitted.

A non-photographing area corresponding to a virtual viewpoint will be described with reference to FIG. 7. FIG. 7 shows the relationship between the cross-sectional view of the stadium shown in FIG. 6 (b) viewed from the horizontal direction and the virtual viewpoint. The virtual viewpoint identified from the viewpoint information has a field of view 705 in the direction of looking up at the ground from the position 701 in the ground. The region 703 is a portion of the region 505 of FIG. 6B included in the visual field 705. That is, the region 703 is a region that is within the field of view of the virtual viewpoint and is a part of the region included in at least one of the imaging ranges of the imaging device 108, and the texture can be obtained. The region 702 is a portion of the region 504 of FIG. 6 (b) included in the visual field 705. That is, the region 702 is a region within the field of view of the virtual viewpoint, but is not included in the imaging range of any of the imaging devices 108, and a texture cannot be obtained. The portion of the region 504 that overlaps the region 703 when viewed from the virtual viewpoint is excluded from the region 702. The region 704 is a region that does not overlap with the region 703 when viewed from a virtual viewpoint, and is a region in which a three-dimensional model of the subject does not exist. The non-photographed area updated by the process of S408 is an area including the area 704 and the area 702. In this way, by setting the non-shooting area, which is the target area for arranging the virtual advertisement, according to the virtual viewpoint, the subject included in the shooting range of the shooting device 108 is shielded by the virtual advertisement in the virtual viewpoint image. The risk can be further reduced.

Next, the details of the process relating to the placement of the advertisement in S307 of FIG. 3 will be described with reference to FIG. In S801, the candidate determination unit 210 determines an area that is a candidate for the placement of the virtual advertisement in the three-dimensional coordinate system from the non-photographing area specified by the area identification unit 208. When inserting a virtual advertisement in a virtual viewpoint image, the virtual advertisement is inserted at a position and orientation compatible with the subject photographed by the photographing device 108, thereby giving a sense of discomfort to the viewer viewing the virtual viewpoint image. Can be reduced. Therefore, the candidate determination unit 210 sets a line or surface substantially perpendicular to or substantially parallel to the surface of the 3D model of the subject corresponding to the foreground or background, the peripheral position of the 3D model of the subject, and the like as candidates for arranging the virtual advertisement. decide. The details of the process of determining the placement candidate will be described later with reference to FIG.

In S802, the placement determination unit 212 determines the placement of the virtual advertisement in the three-dimensional coordinate system according to the placement candidates determined by the candidate determination unit 210. The placement determination unit 212 may preferentially place the virtual advertisement in the area where the advertising effect is high among the decided placement candidates. The area having a high advertising effect is, for example, an area that is displayed for a long time in the virtual viewpoint image, an area that is frequently displayed, an area that is displayed in a large area, and the like. Further, the placement determination unit 212 may place the virtual advertisement in an area where the higher the advertisement fee set for the virtual advertisement is, the higher the advertising effect is. The details of the process of determining the placement of the virtual advertisement will be described later with reference to FIG.

In S803, the advertisement model generation unit 213 generates the model information of the virtual advertisement. Specifically, the advertisement model generation unit 213 generates information for specifying a three-dimensional model representing the shape of the virtual advertisement and information regarding the position of the virtual advertisement determined by the placement determination unit 212. In S804, the advertisement texture determination unit 214 maps the advertisement data stored in advance in the auxiliary storage device 104 as a texture to the three-dimensional model of the virtual advertisement.

Next, with reference to FIG. 9, the details of the process of determining the placement candidate of the virtual advertisement in S801 of FIG. 8 will be described. The placement candidates are initialized at the start of the process of FIG. 8, and the placement candidates are updated in the process of FIG. Then, the placement candidate finally determined after the update is performed becomes the target area for placing the virtual advertisement.

In S901, the candidate determination unit 210 acquires the foreground and background model information. The model information acquired here includes information for identifying a three-dimensional model of the subject, information regarding the position of the subject, and information regarding the texture. In S902, the candidate determination unit 210 determines whether or not the processing of S903 to S912 has been performed on the model information of each subject acquired in S901. If there is model information of an unprocessed subject, one of them is selected as a processing target, and the process proceeds to S903. If there is no model information of the unprocessed subject, the process proceeds to S913.

In S903, the candidate determination unit 210 determines whether or not the subject to be processed is a real advertisement (for example, a signboard) existing in the real space, and if it is a real advertisement, it moves to S904, and if it is not a real advertisement, it moves to S904. Move to S905. In S904, the candidate determination unit 210 adds a position vertically away from the actual advertisement in the three-dimensional coordinate system to the placement candidate of the virtual advertisement, and returns to S902. In the virtual viewpoint image, the virtual advertisement is inserted at a position vertically away from the actual advertisement such as directly above or directly below the actual advertisement, and the advertisement content of the virtual advertisement and the actual advertisement are related to synergistically improve the advertising effect. be able to.

In S905, the candidate determination unit 210 determines whether or not the subject to be processed is the audience seat, and if it is the audience seat, it moves to S906, and if it is not the audience seat, it moves to S908. In S906, the candidate determination unit 210 adds a position vertically away from the boundary between the spectator seat and the field to the placement candidate, and moves to S907. In S907, the candidate determination unit 210 adds the surface of the outer wall of the spectator seat to the placement candidate, and returns to S902. In the present embodiment, since the photographing device 108 is installed in the stadium, the outer wall of the audience seats is not included in the photographing range by the photographing device 108, and the texture corresponding to the surface of the outer wall cannot be obtained.

In S908, the candidate determination unit 210 determines whether or not the subject to be processed is a roof, and if it is a roof, it moves to S909, and if it is not a roof, it moves to S910. In S909, the candidate determination unit 210 adds the outer surface of the roof to the placement candidates and returns to 902. The outer surface of the roof, like the outer wall of the audience seats, is not included in the shooting range of the shooting device 108.

In S910, the candidate determination unit 210 determines whether or not the subject to be processed is a field, and if it is a field, it moves to S911, and if it is not a field, it returns to S902. In S911, the candidate determination unit 210 adds the surface of the field to the placement candidate and moves to S912. In S912, the candidate determination unit 210 adds a surface perpendicular to the line drawn on the field in the three-dimensional coordinate system to the placement candidate, and returns to S902.

In S913, the gazing point acquisition unit 209 acquires the gazing point position according to the photographing direction of the plurality of photographing devices 108. The candidate determination unit 210 adds a region corresponding to a position vertically away from the gazing point in the three-dimensional coordinate system to the placement candidate, and moves to S914. In the virtual viewpoint image, the area around the gazing point has high image quality and is an area that the viewer pays particular attention to. Therefore, the advertising effect can be improved by inserting the virtual advertisement at a position vertically away from the gazing point, such as directly below or above the gazing point.

In S914, the candidate determination unit 210 acquires the non-photographed area specified by the area identification unit 208 and moves to S915. In S915, the candidate determination unit 210 determines only the placement candidates existing in the non-photographing area among the placement candidates added by the processing of S903 to S912 as the final placement candidates, and ends the processing.

FIG. 10 shows an example of placement candidates for virtual advertisements. In this example, placement candidates are determined based on the audience seats 501, the field 502, the actual advertisement 1002, and the gazing point 503. Candidate 1001 is the surface of the outer wall of the audience seat 501. Candidate 1004 is the outer surface of the roof of the auditorium 501. Candidate 1005 is a horizontal plane at ground level outside the stadium. Candidate 1006 is a horizontal plane at ground level under the spectators' seats. Candidate 1007 is a surface vertically separated from the boundary between the audience seats and the field. Candidate 1008 is a surface vertically away from the actual signboard, which is the subject included in the image captured by the photographing apparatus 108. Candidate 1009 is a plane away from the gazing point in the vertical direction.

Although the virtual advertisement has a thick shape in FIG. 10, the virtual advertisement is not limited to this, and the virtual advertisement may be a planar object having no thickness. For example, when the virtual advertisement is placed on the candidate 1001 which is the surface of the outer wall of the audience seat 501, the advertisement may be mapped as the texture of the outer wall.

Here, Candidate 1007, Candidate 1008, and Candidate 1009 are areas corresponding to the ground. Although FIG. 10 shows a subject corresponding to the background and an arrangement candidate determined based on the gazing point, the arrangement candidate may be determined based on a subject corresponding to the foreground such as a player or a ball. For example, a virtual object for displaying information about the player may be placed at a position vertically away from the player. Further, FIG. 10 shows a case where the virtual object has a surface shape, but the present invention is not limited to this, and the virtual object may have a three-dimensional shape such as a sphere or a rectangular parallelepiped.

Next, with reference to FIG. 11, the details of the process of determining the placement of the virtual advertisement in S802 of FIG. 8 will be described. In the following description using FIG. 11, it is assumed that the generated virtual viewpoint image is a moving image. In S1101, the arrangement determination unit 212 acquires the advertisement data of the virtual advertisement stored in advance in the auxiliary storage device 104. When there are a plurality of virtual advertisements inserted in the virtual viewpoint image, a plurality of advertisement data are acquired.

In S1102, the placement determination unit 212 determines whether or not the processing of S1103 to S1108 has been performed on each advertisement data acquired in S1101. If there is unprocessed advertisement data, one of them is selected as the processing target and the process proceeds to S1103. If there is no unprocessed advertisement data, the process ends.

In S1103, the placement determination unit 212 determines whether or not the virtual advertisement to be processed has been placed in the generation of the virtual viewpoint image of the previous frame. If it has already been placed, it moves to S1105, and if it has not been placed, it moves to S1104. In S1104, the placement determination unit 212 determines the placement of the virtual advertisement according to the placement candidates determined by the candidate determination unit 210. When there are a plurality of placement candidates determined by the candidate determination unit 210, the placement determination unit 212 arranges, for example, a virtual advertisement in an area having the highest advertising effect among the placement candidates. However, the method of determining the area where the virtual advertisement is placed from the placement candidates is not limited to this.

In S1105, the arrangement determination unit 212 acquires the viewpoint information held by the viewpoint acquisition unit 211. In S1106, the arrangement determination unit 212 determines whether or not the virtual viewpoint is moving. If the virtual viewpoint is moving, that is, if the virtual viewpoint in the frame to be processed and the virtual viewpoint in the previous frame do not match, the process moves to S1108, and if the virtual viewpoint is not moving, the process moves to S1107.

In S1107, the placement determination unit 212 updates the placement of the virtual advertisement. For example, when the movement of the virtual viewpoint is stopped, if the area of the placement candidates having a high advertising effect changes with the movement of the virtual viewpoint, the placement determination unit 212 may change the placement of the virtual advertisement. .. Further, for example, the arrangement determination unit 212 may change the direction of the virtual advertisement in the three-dimensional coordinate system as the virtual viewpoint moves. In S1108, the placement determination unit 212 ends the process without changing the placement of the virtual advertisement from the placement in the previous frame. That is, the placement of the virtual advertisement is not updated while the virtual viewpoint is moving.

An example of the placement of a virtual advertisement will be described with reference to FIGS. 12 and 13. FIG. 12 shows an example in which the orientation of the virtual advertisement in the three-dimensional coordinate system changes according to the movement of the virtual viewpoint according to the process of FIG. The virtual viewpoint moves continuously with the passage of time from the viewpoint 1201 of FIG. 12 (a) to the viewpoint 1203 of FIG. 12 (c). As described with reference to FIG. 11, the direction of the virtual advertisement 1204 does not change when the virtual viewpoint is moving. On the other hand, FIG. 12 (d) corresponds to the next frame of FIG. 12 (c), and the direction of the virtual advertisement changes according to the stop of the movement of the virtual viewpoint. In this way, by not changing the direction of the advertisement while the virtual viewpoint is moving, it is possible to reduce the discomfort given to the viewer of the virtual viewpoint image. Further, by rearranging the virtual advertisement in the direction corresponding to the virtual viewpoint after the movement of the virtual viewpoint is stopped, the virtual advertisement can be easily noticed by the viewer and the advertising effect can be improved. As described above, the arrangement determination unit 212 may determine the orientation of the virtual object in the three-dimensional coordinate system based on the viewpoint information acquired by the viewpoint acquisition unit 211.

In the example of FIG. 12, the orientation of the virtual advertisement is determined so that the predetermined surface on which the advertisement is displayed is substantially horizontal or substantially perpendicular to the ground in the three-dimensional coordinate system. This makes it easier for the viewer to grasp the horizontal direction and the vertical direction in the virtual viewpoint image. Further, in the example of FIG. 12, the virtual advertisement is arranged in the direction in which the advertisement content is more easily seen from the virtual viewpoint, out of the horizontal orientation and the vertical orientation. The direction in which the advertisement content is easy to see is the direction in which the angle formed by the line-of-sight direction of the virtual viewpoint and the surface of the virtual advertisement is close to a right angle.

In addition, in order to further enhance the advertising effect of the virtual advertisement, the direction of the virtual advertisement may be changed even while the virtual viewpoint is moving. FIG. 13 shows an example in this case. The virtual viewpoint continuously moves from the viewpoint 1201 in FIG. 13 (a) to the viewpoint 1203 in FIG. 13 (c), but the direction in which the advertisement content is easily seen from the virtual viewpoint changes in the viewpoint 1202 in FIG. 12 (b). At this point, the direction of the virtual advertisement is changed.

In the above description using FIG. 11, it is assumed that the arrangement of the virtual advertisement changes according to the change of the virtual viewpoint, but the present invention is not limited to this. That is, the arrangement determination unit 212 may determine a fixed position in the three-dimensional coordinate system that does not change according to the viewpoint information acquired by the viewpoint acquisition unit 211 as the position of the virtual advertisement. As a result, it is possible to reduce the discomfort felt by the viewer of the virtual viewpoint image because the positional relationship between the virtual advertisement and the subject included in the captured image in the three-dimensional coordinate system changes with the movement of the virtual viewpoint.

FIG. 14 shows an example of a virtual viewpoint image generated by the rendering processing unit 215 according to the determination of the arrangement of the virtual advertisement by the arrangement determination unit 212. FIG. 14A is an example of a virtual viewpoint image corresponding to a virtual viewpoint designated to look up from the ground to the ground. A player 1400, an actual advertisement 1002, and a spectator seat 501 are displayed as subjects included in the image captured by the photographing device 108. The area below the boundary 1403 between the field and the audience seats is the underground area 1405 corresponding to the outside of the shooting range by the shooting device 108. In the example of FIG. 14, since the field is transparently displayed, it is displayed as if the player 1400 actually touching the field is floating.

The area 1405 corresponding to the outside of the shooting range is a black image, and the virtual advertisement 1204 is displayed in this area 1405. By displaying a virtual object for providing predetermined information to the user, such as a virtual advertisement 1204 having character information, in the area 1405 which is conventionally a black image and only indicates that it is out of the shooting range. , The limited area in the virtual viewpoint image can be effectively used. It should be noted that the area other than the virtual advertisement 1204 in the area 1405 is not limited to the black image, and an image of a predetermined color or pattern may be assigned.

As in the example of FIG. 14A, the virtual advertisement 1204 may be displayed in an area not adjacent to the area corresponding to the shooting range (the area above the boundary 1403). By doing so, it becomes easy for the viewer of the virtual viewpoint image to distinguish between the area corresponding to the shooting range and the area corresponding to the outside of the shooting range.

Further, in the present embodiment, when the virtual viewpoint image includes a region corresponding to the shooting range and a region corresponding to the outside of the shooting range, the virtual viewpoint image is virtual to the region corresponding to the outside of the shooting range. The case of displaying an advertisement will be described. However, the present invention is not limited to this, and the advertisement may be displayed outside the virtual viewpoint image. For example, in the display area of the display unit of the terminal device 109, an advertisement area in which the advertisement is displayed may exist outside the area in which the virtual viewpoint image is displayed. Then, the shape of the advertising area, the display content of the advertisement displayed in the advertising area, whether or not the advertisement is displayed in the advertising area, and the like change according to the virtual viewpoint related to the generation of the virtual viewpoint image. You may.

FIG. 14B is an example of a virtual viewpoint image corresponding to a virtual viewpoint designated to look down on the stadium from the sky. The virtual advertisement "ADV1" is displayed in the area 1406 corresponding to the roof of the stadium where the 3D model exists but the texture does not exist, and the virtual advertisement "ADV2" is displayed in the area 1407 corresponding to the outside of the stadium where the 3D model does not exist. It is displayed.

In the present embodiment, the case where the virtual advertisement is inserted only in the area corresponding to the outside of the shooting range in the virtual viewpoint image has been mainly described, but the present invention is not limited to this, and the area corresponds to the shooting range. Virtual ads may be inserted in both areas corresponding to the outside. Even with such a configuration, it is possible to reduce the possibility that the subject included in the shooting range is obscured by the virtual object as compared with the case where all the virtual advertisements are inserted in the corresponding area within the shooting range.

Further, in the present embodiment, the arrangement determination unit 212 determines the arrangement of the virtual advertisement in the virtual three-dimensional coordinate system, and the rendering processing unit 215 inserts the virtual advertisement in the virtual viewpoint image based on the determined arrangement. The case of determining the insertion area has been described. However, the information processing device 100 is not limited to this, and for example, the information processing apparatus 100 generates a virtual viewpoint image in which a virtual advertisement is not inserted, and a region corresponding to the outside of the shooting range in the virtual viewpoint image is set as an external parameter related to shooting by the shooting device 108. It may be specified based on internal parameters. Then, the information processing apparatus 100 may determine the specified area in the virtual viewpoint image as an insertion area for inserting a virtual advertisement.

As described above, the information processing apparatus 100 according to the present embodiment determines the insertion region of the virtual viewpoint image into which the virtual object not included in the images captured by the plurality of imaging devices 108 is inserted. Specifically, data based on shooting from different directions by the plurality of shooting devices 108 is acquired, and viewpoint information according to the designation of the virtual viewpoint related to the generation of the virtual viewpoint image based on the data is acquired. .. Then, the information processing apparatus 100 determines an region in the virtual viewpoint image corresponding to the region outside the imaging range by the plurality of imaging devices 108 as an insertion region into which the virtual object is inserted. According to such a configuration, when a virtual object is inserted into the virtual viewpoint image, it is possible to reduce the possibility that the subject included in the shooting range by the shooting device 108 is obscured by the virtual object in the virtual viewpoint image. can.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a recording medium readable by a computer.

10 Image processing system 100 Information processing device 108 Imaging device

Claims (22)

A data acquisition means for acquiring image data corresponding to an image taken based on shooting by a plurality of shooting devices that shoot a shooting range from different positions, and a data acquisition means.
The plurality of photographs taken from a plurality of candidate areas defined as areas that do not block a predetermined subject included in the virtual viewpoint image generated based on the image data acquired by the data acquisition means and the viewpoint information indicating the virtual viewpoint. The first determination means for determining a specific area corresponding to the outside of the shooting range by the device, and
It is characterized by having a second determining means for determining the position of the virtual object so that the virtual object that does not actually exist in the specific area is displayed at the position corresponding to the specific area in the virtual viewpoint image. Information processing device. The information processing apparatus according to claim 1, wherein the specific area is an area in which a subject photographed by the plurality of photographing devices does not exist. The specific area is a region inside a subject photographed by the plurality of photographing devices and / or a region of a surface of the subject which is not photographed by the plurality of photographing devices. The information processing apparatus according to 1. The information processing apparatus according to any one of claims 1 to 3, wherein the specific area is a region not adjacent to the photographing range. It has a parameter acquisition means for acquiring parameters related to imaging by the plurality of imaging devices.
One of claims 1 to 4, wherein the first determination means determines the specific area based on the result of specifying the imaging range based on the parameters acquired by the parameter acquisition means. The information processing device described in the section. The information processing apparatus according to any one of claims 1 to 5, wherein the second determination means determines the position of the virtual object in the virtual viewpoint image. The information processing apparatus according to any one of claims 1 to 5, wherein the second determination means determines the position of the virtual object in a three-dimensional space. The information processing apparatus according to any one of claims 1 to 7, wherein the specific area is an area that does not change according to the virtual viewpoint. The information processing apparatus according to any one of claims 1 to 7, wherein the specific area is determined based on the image data acquired by the data acquisition means and the virtual viewpoint. The information processing apparatus according to any one of claims 1 to 9, wherein the orientation of the virtual object is determined based on the virtual viewpoint. The orientation of the virtual object is determined so that the orientation of the virtual object does not change while the virtual viewpoint is moving and the orientation of the virtual object changes after the movement of the virtual viewpoint is stopped. The information processing apparatus according to claim 10. The invention according to any one of claims 1 to 11, wherein the orientation of the virtual object is determined so that a predetermined surface of the virtual object is substantially horizontal or substantially perpendicular to the ground in a three-dimensional space. Information processing device. The information processing apparatus according to any one of claims 1 to 12, wherein the specific area is an underground area. 6. Information processing equipment. The information processing according to any one of claims 1 to 12, wherein the specific region is a region at a position vertically separated from a subject photographed by the plurality of photographing devices in a three-dimensional space. Device. The information processing device according to any one of claims 1 to 15, wherein the virtual object is an object for displaying an advertisement on the virtual viewpoint image. It has an information acquisition means for acquiring the viewpoint information, and has
The virtual viewpoint image in which the virtual object is displayed at a position determined by the second determination means is based on the image data acquired by the data acquisition means and the viewpoint information acquired by the information acquisition means. The information processing apparatus according to any one of claims 1 to 16, further comprising a generation means for generating the information. It has an information acquisition means for acquiring the viewpoint information, and has
A generation means for generating a virtual viewpoint image in which the virtual object is not inserted based on the image data acquired by the data acquisition means and the viewpoint information acquired by the information acquisition means.
Output to output the virtual viewpoint image generated by the generation means and the information for specifying the position determined by the second determination means to the device for generating the virtual viewpoint image into which the virtual object is inserted. The information processing apparatus according to any one of claims 1 to 16, further comprising means. A data acquisition process that acquires image data corresponding to images taken based on shooting by multiple shooting devices that shoot the shooting range from different positions, and
The plurality of photographs taken from a plurality of candidate areas defined as areas that do not block a predetermined subject included in the virtual viewpoint image generated based on the image data acquired in the data acquisition step and the viewpoint information indicating the virtual viewpoint. The first determination step of determining a specific area corresponding to the outside of the shooting range by the device , and
It is characterized by having a second determination step of determining the position of the virtual object so that the virtual object that does not actually exist in the specific area is displayed at the position corresponding to the specific area in the virtual viewpoint image. Information processing method. The information processing method according to claim 19 , wherein the specific area is an area in which a subject photographed by the plurality of photographing devices does not exist. The specific area is a region inside a subject photographed by the plurality of photographing devices and / or a region of a surface of the subject which is not photographed by the plurality of photographing devices. The information processing method according to 19 . A program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 18.

Images