JP2021179800A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To easily recognize relationship between viewpoints corresponding to a captured image and a virtual viewpoint image, respectively, in an image content including the captured image obtained by a moving imaging apparatus and the virtual viewpoint image.SOLUTION: A viewpoint operation UI 350 specifies a position of an in-vehicle imaging apparatus 200 moving in an imaging area in which images are captured by a plurality of imaging apparatuses 100 from different positions, by using data based on imaging by the imaging apparatuses 100. The viewpoint operation UI 350 determines a position of a virtual viewpoint corresponding to a virtual viewpoint image generated on the basis of the images obtained by the imaging apparatuses 100, in accordance with the specified position of the in-vehicle imaging apparatus 200.SELECTED DRAWING: Figure 1

Description

The present invention relates to a virtual viewpoint image generated based on a captured image.

There is a technique in which a plurality of imaging devices are installed at different positions to perform synchronous imaging from multiple viewpoints, and a virtual viewpoint image in which the viewpoint can be arbitrarily changed is generated using the plurality of images obtained by the imaging. For example, by generating a virtual viewpoint image according to a viewpoint designated by the user based on a plurality of images of a competition such as soccer or basketball, the user can watch the competition from various viewpoints.

Patent Document 1 describes a technique in which a plurality of photographing devices are arranged so as to surround a field used for soccer or the like, and a virtual viewpoint image is generated using images photographed by the plurality of photographing devices.

Japanese Unexamined Patent Publication No. 2017-21128

The application destination of the technology for generating a virtual viewpoint image is not limited to soccer and basketball, and for example, this technology can be applied to motor sports and the like. On the other hand, in motor sports, an in-vehicle camera may be used to provide a viewer with a realistic image. Therefore, in order to provide the viewer with a new video experience, it is conceivable to generate image content including both the image of the in-vehicle camera and the virtual viewpoint image. However, depending on the method of determining the viewpoint corresponding to the virtual viewpoint image, it may be difficult for the viewer to recognize the relationship between the image of the in-vehicle camera included in the image content and the viewpoint corresponding to each of the virtual viewpoint images, which may cause confusion. ..

The present invention has been made in view of the above problems, and in the image content including the captured image and the virtual viewpoint image acquired by the moving photographing device, the viewpoint corresponding to each of the captured image and the virtual viewpoint image. The purpose is to make the relationship easily recognizable.

In order to solve the above problems, the information processing apparatus according to the present invention has, for example, the following configuration. That is, the position of the mobile photographing device for acquiring the captured image included in the image content to be generated and moving within the photographing area photographed from different positions by a plurality of photographing devices is described above. Based on a specific means for specifying using data based on shooting by a shooting device included in a plurality of shooting devices, and a plurality of images acquired by the plurality of shooting devices, which are virtual viewpoint images included in the image content. It has a determination means for determining the position of the virtual viewpoint corresponding to the generated virtual viewpoint image to a position corresponding to the position of the mobile photographing apparatus specified by the specific means.

According to the present invention, it is possible to easily recognize the relationship between the viewpoints corresponding to each of the photographed image and the virtual viewpoint image in the image content including the photographed image and the virtual viewpoint image acquired by the moving photographing device.

It is a figure which shows the configuration example of an image processing system. It is a figure which shows the race ground as an example of a shooting area. It is a figure which shows an example of the vehicle which installed the photographing apparatus. It is a figure which shows the functional configuration example of an image generator. It is a figure which shows the functional structure example of the viewpoint operation UI. It is a figure which shows an example of the hardware composition of a device. It is a figure which shows the example of the vehicle identification table. It is a figure which shows the example of the calibration table. It is a flowchart which shows the example of the process which concerns on the acquisition of the calibration information of an in-vehicle photographing apparatus. It is a flowchart which shows the example of the process which concerns on the generation of a virtual viewpoint image, and the conversion of a calibration information. It is a figure which shows the example of the photographed image, the foreground image, the background image, and a marker detection result. It is a figure which shows the example of the photographed image, the foreground image, the background image, and a marker detection result. It is an image diagram which shows the position on the 3D model of the marker detected from the photographed image. It is an image diagram which shows the relationship of a plurality of coordinate systems. It is a flowchart which shows the example of the process which concerns on the determination of a virtual viewpoint and the instruction of switching an image. It is a flowchart which shows the example of the process which concerns on the determination of a virtual viewpoint and the instruction of switching an image. It is a figure which shows the example of the time change of the distance between an in-vehicle photographing apparatus and a virtual viewpoint. It is a flowchart which shows the detailed flow of the processing of a tracking part. It is a flowchart which shows the processing flow in the modification.

[System configuration]
FIG. 1 is a diagram showing a configuration example of the image processing system 1. The image processing system 1 is a system that provides a viewer with image content that is displayed by switching between a captured image and a virtual viewpoint image. The virtual viewpoint image is an image representing a view from a designated virtual viewpoint, which is generated based on a plurality of images taken by a plurality of photographing devices and a designated virtual viewpoint. The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images and the like are also included in the virtual viewpoint image. Further, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

The image processing system 1 includes a photographing system 110, an in-vehicle photographing device 200, a hub 300, a time server 310, a control device 320, a control UI 330, an image generation device 340, a viewpoint operation UI 350, and an output device 360. The photographing system 110 includes a plurality of photographing devices from the photographing device 100-1 to the photographing device 100-12 for photographing the photographing area from a plurality of directions. In the present embodiment, the number of photographing devices is 12, but the number of photographing devices included in the photographing system 110 is not limited to this. In the following, when each photographing device is not particularly distinguished, it is referred to as a photographing device 100.

FIG. 2 shows a motor sports race field 130, which is an example of a photographing area photographed by the photographing system 110 in the present embodiment. A plurality of photographing devices 100 are installed at different positions so as to surround the photographing area, and image is taken in synchronization. The plurality of photographing devices 100 may not be installed over the entire circumference of the photographing area, and may be installed only in a part around the photographing area depending on the limitation of the installation place or the like, or may be installed inside the photographing area. It may be installed in. Further, a shooting device having different functions such as a telephoto camera and a wide-angle camera may be installed.

It should be noted that the plurality of photographing devices 100 in the present embodiment are cameras having independent housings and capable of photographing from a single viewpoint. However, the present invention is not limited to this, and two or more photographing devices 100 may be configured in the same housing. For example, a single camera having a plurality of lens groups and a plurality of sensors and capable of photographing from a plurality of viewpoints may be installed as a plurality of photographing devices 100. Further, in the present embodiment, the case where the shooting area is the race field 130 will be described, but the shooting area is not limited to this, and the shooting area is, for example, a stadium where competitions such as soccer and karate are performed, or a stage where a concert or a play is performed. And so on.

The image generation device 340 includes the position and orientation of each imaging device 100 in the coordinate system 120 with the reference position in the race field 130 as the origin, and the focal length of each imaging device 100 for the virtual viewpoint image generation processing described later. A parameter indicating the distance is stored. The parameters related to these shootings are hereinafter referred to as calibration information of the shooting system 110. The calibration information may include other parameters related to shooting. Further, in the present embodiment, it is assumed that the imaging system 110 calibration information is stored in the image generation device 340 in advance, but the image generation device 340 stores the calibration information from other devices such as the imaging system 110 and the control device 320. You may get it.

The hub 300 acquires a plurality of captured images captured by the photographing system 110 and outputs them to the image generation device 340. Here, the time server 310 connected to the hub 300 transmits the time synchronization information to the photographing system 110, and the photographing system 110 transmits the photographed image in synchronization based on the time synchronization information from the time server 310. Each photographing device 100 of the photographing system 110 may be connected to the hub 300 on a one-to-one basis, or the photographing devices 100 may be connected to each other in a daisy chain.

The control device 320 controls the state of the photographing system 110. The control UI 330 is a user interface that enables a user operation for controlling the state of the photographing system 110, and sends an input corresponding to the user operation to the control device 320. Specifically, the control of the state of the shooting system 110 includes time synchronization setting of the shooting system 110, start / stop of shooting, adjustment of zoom and focus, and setting of brightness such as ISO and IRIS. The control device 320 and the control UI 330 may be physically integrated. Further, the control device 320 may be directly connected to the photographing system 110 without going through the hub 300.

The image generation device 340 is based on the calibration information of the photographing system 110 in the coordinate system 120 shown in FIG. 2, the plurality of captured images acquired by the photographing system 110, and the viewpoint information acquired from the viewpoint operation UI 350. Generate a virtual viewpoint image. The generated virtual viewpoint image is output to the viewpoint operation UI 350 and the output device 360. The viewpoint information used to generate the virtual viewpoint image is information indicating the position and direction (line-of-sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter representing a three-dimensional position of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. The content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint. Further, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points. ..

The virtual viewpoint image is generated by, for example, the following method. First, a plurality of images (multiple viewpoint images) are acquired by taking pictures from different directions by the plurality of photographing devices 100. Next, a foreground image in which a foreground area corresponding to a predetermined object such as a race vehicle is extracted and a background image in which a background area other than the foreground area is extracted are acquired from the plurality of viewpoint images. In addition, a foreground model representing the three-dimensional shape of a predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and the background model representing the three-dimensional shape of the background such as a stadium is colored. Texture data is generated based on the background image. Then, a virtual viewpoint image is generated by mapping the texture data to the foreground model and the background model and performing rendering according to the virtual viewpoint indicated by the viewpoint information. However, the method of generating a virtual viewpoint image is not limited to this, and various methods such as a method of generating a virtual viewpoint image by projective transformation of a captured image without using a three-dimensional model can be used.

The viewpoint operation UI 350 is a user interface that enables user operations for operating a virtual viewpoint, and outputs viewpoint information to the image generation device 340 as input according to the user operation. Further, the viewpoint operation UI 350 issues an instruction to the output device 360 to switch between the virtual viewpoint image and the captured image acquired by the in-vehicle imaging device 200. The details of the switching instruction will be described later. The image generation device 340 and the viewpoint operation UI 350 may be physically integrated.

The in-vehicle photographing device 200 is an example of a moving photographing device that moves in the photographing area, and is an in-vehicle camera that is installed in a vehicle traveling in the race field 130 and takes an image while moving together with the vehicle. Then, the in-vehicle photographing device 200 transmits the captured image to the output device 360 via the wireless communication device. A plurality of in-vehicle photographing devices 200 may be installed in one vehicle. Further, the in-vehicle photographing device 200 may be installed in all of a plurality of vehicles participating in the race, or may be installed in only some of the vehicles.

FIG. 3 shows an example of a vehicle in which the in-vehicle photographing device 200 is installed. In this example, the vehicle-mounted photographing device 200 is mounted on the four-wheeled vehicle 230. The image generation device 340 has parameters indicating the position and orientation of the vehicle-mounted image pickup device 200 in the coordinate system 220 with the reference position of the vehicle 230 as the origin, and the focal length of the vehicle-mounted image pickup device 200 for controlling the virtual viewpoint described later. Is remembered. The parameters related to these shootings are hereinafter referred to as calibration information of the in-vehicle shooting device 200. The parameters included in the calibration information of the in-vehicle photographing apparatus 200 are not limited to the above. Further, in the present embodiment, it is assumed that the calibration information of the in-vehicle photographing apparatus 200 is stored in the image generating apparatus 340 in advance, but the image generating apparatus 340 is calibrated from other apparatus such as the photographing system 110 and the control device 320. Information may be obtained. The coordinate system 220 is a three-dimensional coordinate system in which the front of the vehicle 230 is the x-axis, the side is the y-axis, and the upper side is the z-axis. For example, the relative position of the vehicle-mounted photographing apparatus 200 with respect to the origin (0, 0, 0) is represented by coordinates (Xs1, Ys1, Zs1).

Further, the vehicle 230 has a vehicle identification marker 210-1, a marker 210-2, and a marker for uniquely identifying the vehicle or specifying the shape of the vehicle from the photographed image acquired by the photographing system 110. 210-3 and a marker 210-4 are attached. In the following, when these markers are not particularly distinguished, they are referred to as markers 210. The marker 210 is provided so that a plurality of markers 210 can be simultaneously photographed from the photographing apparatus 100 of the photographing system 110. In the example of FIG. 3, four markers 210 are attached to the vehicle 230, but more markers 210 may be attached so that the orientation and shape of the vehicle can be accurately specified. Further, in order to improve the specific accuracy of the vehicle, the patterns of the respective markers 210 are different, but some or all of the markers 210 may have the same pattern. Further, although the marker 210 is used as a specific geometric pattern in the example of FIG. 3, an advertisement logo attached to the vehicle, a feature point of the vehicle, or the like may be used as the marker 210 for vehicle identification. The image generation device 340 also stores in advance the position of each marker 210 in the coordinate system 220. For example, the coordinates of the positions of the four markers 210 are represented by (Xm1, Ym1, Zm1), (Xm2, Ym2, Zm2), (Xm3, Ym3, Zm3), and (Xm4, Ym4, Zm4), respectively.

The output device 360 receives the input of the virtual viewpoint image from the image generation device 340 and the input of the captured image from the vehicle-mounted photographing device 200. Then, the output device 360 outputs while switching between the virtual viewpoint image and the captured image based on the switching instruction from the viewpoint operation UI 350. The output destination of the image from the output device 360 is, for example, a storage device (not shown), a display device, an image distribution network, or the like. By outputting the image from the output device 360, the image content including the virtual viewpoint image and the captured image is generated, and finally the image content is provided to the viewer.

In the present embodiment, in the image content output by the output device 360, the virtual viewpoint image and the captured image of the in-vehicle photographing device 200 are switched and displayed, but the display content of the image content is limited to this. Not done. For example, the output device 360 may simultaneously output the captured image of the vehicle-mounted imaging device 200 and the virtual viewpoint image to generate image content in which these images are displayed at the same time. Further, the output device 360 may output the image content generated by synthesizing the captured image of the vehicle-mounted photographing device 200 and the virtual viewpoint image. Further, the images included in the image contents to be generated are not limited to the virtual viewpoint image and the captured image of the in-vehicle photographing device 200. The image content to be generated may include, for example, an image acquired by a photographing device 100 or another camera installed in a race track, or may include information such as a telop other than the image. Further, the switching instruction may be input from a device other than the viewpoint operation UI 350.

Further, the output device 360 also has a function of adjusting the input delay difference between the virtual viewpoint image and the captured image of the vehicle-mounted photographing device 200. While there is a delay in the image processing from shooting by the shooting system 110 to the generation of the virtual viewpoint image, there is also a delay in the transmission processing of the shot image from the in-vehicle shooting device 200 to the output device 360. The output device 360 has an image buffer for adjusting the difference between these delays, and can synchronize the virtual viewpoint image and the captured image that are output while switching.

The configuration of the image processing system 1 is not limited to the example shown in FIG. For example, the image generation device 340 and the control device 320 may be integrally configured. Further, the captured image acquired by the in-vehicle imaging device 200 may be input to the image generation device 340, and the image generation device 340 may synthesize the captured image and the virtual viewpoint image and output the captured image to the output device 360.

FIG. 4 shows an example of the functional configuration of the image generation device 340. The image generation device 340 includes an image processing unit 3401, an image storage unit 3402, an object information input unit 3403, a calibration information input unit 3404, an object identification unit 3405, a coordinate conversion unit 3406, a viewpoint input unit 3407, and an information storage unit 3408. Have. Each functional unit is connected via an internal bus 3409, and bidirectional communication is possible.

The image processing unit 3401 generates a 3D model which is data representing the three-dimensional shape of an object such as a vehicle based on the photographed image acquired from the photographing system 110 and the calibration information of the photographing system 110, and the image storage unit described later. Store in 3402. Further, the image processing unit 3401 reads a 3D model from the image storage unit 3402 and renders the 3D model based on the viewpoint information acquired from the viewpoint operation UI 350 to generate a virtual viewpoint image. The generated virtual viewpoint image is output to the output device 360 and the viewpoint operation UI 350. The image processing unit 3401 performs a series of processing from the generation of the 3D model to the rendering and the output of the virtual viewpoint image to the outside within a predetermined time. For example, when a virtual viewpoint image is used in live broadcasting, this series of processing is processed within a time of a period V defined by a predetermined frame rate. The image storage unit 3402 stores a 3D model of the object, texture information of the object necessary for rendering, and calibration information.

The object information input unit 3403 acquires the vehicle identification table 700 from the outside and stores it in the information storage unit 3408. FIG. 7 shows an example of the vehicle identification table 700. The vehicle identification table 700 describes the pattern of the vehicle identification marker 210 assigned to the vehicle A and the coordinates of each marker 210 in the coordinate system 220 with respect to the vehicle A. When there are a plurality of vehicles in the race track 130, the pattern of the marker 210 is different for each vehicle in order to improve the identification accuracy of the vehicles. However, the present invention is not limited to this, and a part of the markers 210 given to the plurality of vehicles may have the same pattern. In this case, the combination of the markers 210 is made different for each vehicle, so that the vehicle can be identified.

The calibration information input unit 3404 acquires the calibration table 800 in which the calibration information of the in-vehicle photographing apparatus 200 is described from the outside and stores it in the information storage unit 3408. FIG. 8 shows an example of the calibration table 800. In the calibration table 800, in addition to the position coordinates of the in-vehicle photographing apparatus 200 installed in the vehicle A, vector information indicating the direction of the in-vehicle photographing apparatus 200 is described. In this example, the vector information indicating the optical axis direction of the in-vehicle photographing apparatus 200 is described, but the information indicating the inclination of the in-vehicle photographing apparatus 200 may be described. Further, the calibration table 800 may contain information on the focal length and brightness of the in-vehicle photographing apparatus 200.

The object identification unit 3405 identifies vehicles in the photographing area based on the 3D model of the object stored in the image storage unit 3402 and the vehicle identification table 700 stored in the information storage unit 3408, and positions each vehicle. And specify the orientation. Details of the process for specifying the position and orientation will be described later. The coordinate conversion unit 3406 specifies the relationship between the coordinate system 220 corresponding to the vehicle identified by the object identification unit 3405 and the coordinate system 120 corresponding to the race field 130. Then, the coordinate conversion unit 3406 changes the calibration information of the in-vehicle photographing apparatus 200 included in the calibration table 800 stored in the information storage unit 3408 from the coordinate value corresponding to the coordinate system 220 to the coordinate value corresponding to the coordinate system 120. Convert. The details of the coordinate value conversion process will be described later.

The viewpoint input unit 3407 acquires viewpoint information according to the designation of the virtual viewpoint from the viewpoint operation UI 350, and inputs the viewpoint information to the image processing unit 3401. Acquisition of viewpoint information is performed every cycle V. Further, the viewpoint input unit 3407 outputs the calibration information of the in-vehicle photographing apparatus 200 converted by the coordinate conversion unit 3406 to the viewpoint operation UI 350. The information storage unit 3408 stores the vehicle identification table 700 and the calibration table 800.

FIG. 5 shows an example of a functional configuration of the viewpoint operation UI 350. The viewpoint operation UI 350 includes a viewpoint output unit 3501, a viewpoint generation unit 3502, a tracking unit 3503, a switching instruction unit 3504, a viewpoint storage unit 3505, an operation input unit 3506, and a display output unit 3507. These functional units are connected via the internal bus 3508, and bidirectional communication is possible.

The viewpoint output unit 3501 outputs the viewpoint information generated by the viewpoint generation unit 3502 to the image generation device 340 for each period V. Further, the viewpoint output unit 3501 acquires the converted calibration information of the in-vehicle photographing apparatus 200 from the image generation device 340 and stores it in the viewpoint storage unit 3505. The viewpoint generation unit 3502 generates viewpoint information based on the operation information input via the operation input unit 3506. The operation information includes information indicating parallel movement of the virtual viewpoint in the vertical and horizontal directions, change of the direction in the pan and tilt directions, change of the focal length, and instruction of the movement speed of the virtual viewpoint. The viewpoint generation unit 3502 generates viewpoint information indicating the changed virtual viewpoint obtained by making changes according to the operation information to the current virtual viewpoint. The tracking unit 3503 determines the distance between the current position of the in-vehicle imaging device 200 indicated by the calibration information stored in the viewpoint storage unit 3505 and the current position of the virtual viewpoint indicated by the viewpoint information output by the viewpoint output unit 3501. Based on this, the next virtual viewpoint is determined.

The switching instruction unit 3504 instructs the output device 360 to switch from the virtual viewpoint image to the captured image of the vehicle-mounted photographing device 200, or to switch from the captured image of the vehicle-mounted photographing device 200 to the virtual viewpoint image. The viewpoint storage unit 3505 stores the calibration information of the vehicle-mounted photographing apparatus 200 for each cycle V via the viewpoint output unit 3501. The operation input unit 3506 accepts a user operation for designating a virtual viewpoint, and outputs operation information corresponding to the user operation to the viewpoint generation unit 3502. Further, the operation input unit 3506 accepts a user operation for instructing image switching, and outputs an input corresponding to the user operation to the image switching instruction unit 3504. The display output unit 3507 displays a virtual viewpoint image acquired from the image generation device 340. The user can perform the operation while visually confirming the displayed virtual viewpoint image.

[Hardware configuration]
FIG. 6 shows an example of the hardware configuration of the viewpoint operation UI 350. The hardware configurations of the control device 320, the control UI 330, and the image generation device 340 are the same as the configurations of the viewpoint operation UI 350 described below. The viewpoint operation UI 350 includes a CPU 3510, a ROM 3511, a RAM 3512, an auxiliary storage device 3513, a display unit 3514, an operation unit 3515, a communication I / F 3516, and a bus 3517.

The CPU 3510 realizes each function of the viewpoint operation UI 350 shown in FIG. 4 by controlling the entire viewpoint operation UI 350 using computer programs and data stored in the ROM 3511 and the RAM 3512. The viewpoint operation UI 350 may have one or more dedicated hardware different from the CPU 3510, and the dedicated hardware may execute at least a part of the processing by the CPU 3510. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 3511 stores programs and the like that do not require changes. The RAM 3512 temporarily stores programs and data supplied from the auxiliary storage device 3513, data supplied from the outside via the communication I / F 3516, and the like. The auxiliary storage device 3513 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display unit 3514 is composed of, for example, a liquid crystal display LED or the like, and displays a GUI (Graphical User Interface) for the user to operate the information processing device. The operation unit 3515 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 3510 in response to an operation by the user. The CPU 3510 operates as a display control unit that controls the display unit 3514 and an operation control unit that controls the operation unit 3515. The communication I / F 3516 is used for communication with an external device of the viewpoint operation UI 350. For example, when the viewpoint operation UI 350 is connected to an external device by wire, a communication cable is connected to the communication I / F 3516. When the viewpoint operation UI 350 has a function of wirelessly communicating with an external device, the communication I / F 3516 includes an antenna. The bus 3517 connects each part of the viewpoint operation UI 350 to transmit information.

In the present embodiment, it is assumed that the display unit 3514 and the operation unit 3515 exist inside the viewpoint operation UI 350, but at least one of the display unit 3514 and the operation unit 3515 exists as another device outside the viewpoint operation UI 350. You may.

[Operation flow of image generator]
FIG. 9 is a flowchart showing an example of processing related to acquisition of calibration information of the in-vehicle photographing apparatus 200 by the image generating apparatus 340. The process shown in FIG. 9 is realized by the CPU 3510 of the image generation device 340 expanding the program stored in the ROM 3511 into the RAM 3512 and executing the program. It should be noted that at least a part of the processing shown in FIG. 9 may be realized by one or a plurality of dedicated hardware different from the CPU 3510. The same applies to the processing of the flowchart shown in FIG. 10 described later. The process shown in FIG. 9 is executed in the preset process before starting the generation of the image content. For example, the process of FIG. 9 is started at the timing when a preset instruction is input to the image generation device 340 after the in-vehicle photographing device 200 is mounted on the vehicle 230 or after the mounting position is determined. .. However, the start timing of the process shown in FIG. 9 is not limited to this.

In S501, the image generation device 340 sets the coordinate system 220 with the reference position of the vehicle 230 as the origin. As shown in FIG. 3, the X, Y, and Z axis directions are set with the reference position of the vehicle 230 as the origin. In S502, the image generation device 340 sets the coordinate information of the marker 210 for vehicle identification in the coordinate system 220. The set coordinate information is stored in the vehicle identification table 700 shown in FIG. 7. In S503, the image generation device 340 sets parameters indicating the position and orientation in the coordinate system 220 of the vehicle-mounted photographing device 200 installed in the vehicle 230, and stores them as the calibration table 800 shown in FIG. In the processing of S502 and S503, for example, information indicating the result of the image generation device 340 measuring the shape of the vehicle 230 and the positions of the marker 210 and the in-vehicle photographing device 200 is acquired from the outside, and the coordinate values and the coordinate values are obtained based on the information. It is executed by calculating the parameters.

FIG. 10 is a flowchart showing an example of processing related to generation of a virtual viewpoint image and conversion of calibration information by the image generation device 340. The process shown in FIG. 10 is started at the timing when an instruction to generate a virtual viewpoint image is input to the image generation device 340 after the process shown in FIG. 9 and the image taken by the photographing system 110. However, the start timing of the process shown in FIG. 10 is not limited to this. The process shown in FIG. 10 may be executed in parallel with the shooting by the shooting system 110.

In S601, the image processing unit 3401 acquires a plurality of captured images from the imaging system 110. Specifically, the image processing unit 3401 acquires a plurality of captured images of the same frame (same time) obtained by photographing the plurality of photographing devices 100 in time synchronization. In S602, the image processing unit 3401 extracts a vehicle region as a foreground and a background region other than the foreground region from each captured image. In the present embodiment, as a method of extracting the foreground and the background, the difference in color and brightness for each pixel between the captured image of a certain frame and the captured image one to several frames before is calculated, and the difference is a predetermined value or more. A method of separating an area consisting of pixels as a foreground is adopted. However, the method of separating the foreground and the background is not limited to this. FIG. 11A shows an example of the image 1100 captured by the photographing apparatus 100-1, and FIG. 11B shows an example of the image 1101 in the foreground region and the image 1102 in the background region extracted from the captured image 1100. Similarly, FIG. 12A shows an example of an image 1200 captured by the photographing apparatus 100-2, and FIG. 12B shows an example of an image 1201 in the foreground region and an image 1202 in the background region extracted from the captured image 1200. Is shown. In the case of motor sports, moving vehicles are extracted as objects in the foreground.

In S603, the image processing unit 3401 generates a 3D model of the object based on the extracted foreground image and the calibration information of the photographing system 110, and stores the 3D model in the image storage unit 3402. Since the calibration information of the photographing system 110 corresponds to the coordinate system 120 with respect to the race field 130, the 3D model representing the vehicle in the race field 130 is arranged in the three-dimensional space corresponding to the coordinate system 120. ..

In S604, the viewpoint input unit 3407 acquires viewpoint information indicating the position and orientation of the virtual viewpoint in the coordinate system 120 with respect to the race field 130 from the viewpoint operation UI 350. In S605, the image processing unit 3401 performs rendering processing using the viewpoint information acquired in S604 and the 3D model stored in the image storage unit 3402, and generates a virtual viewpoint image. In S606, the image processing unit 3401 outputs a virtual viewpoint image to the output device 360 and the viewpoint operation UI 350.

In S607, the image generation device 340 determines whether or not to end the generation process of the virtual viewpoint image. For example, when an instruction to end the generation process is input to the image generation device 340, the process of FIG. 10 ends. On the other hand, if the virtual viewpoint image generation process is not completed, the process returns to S601. The processes of S601 to S612 are repeatedly executed every cycle V until it is determined that the virtual viewpoint image generation process is completed. Since the 3D model is stored in the image storage unit 3402 in S603, the virtual viewpoint can be changed later to create a virtual viewpoint image for replay or highlighting.

In S608, the object identification unit 3405 detects the vehicle identification marker 210 from the foreground image extracted in S602. Specifically, the object identification unit 3405 reads out the vehicle identification table 700 stored in the information storage unit 3408 in advance, and detects the marker 210 by performing a marker pattern matching process on the foreground image. FIG. 11C shows an image 1103 as a result of detecting the marker 210 from the image 1101 in the foreground region. FIG. 12C shows an image 1203 as a result of detecting the marker 210 from the image 1201 in the foreground region. In the present embodiment, in order to reduce erroneous detection, it is assumed that a marker having a size smaller than a predetermined value or a marker having a tilt equal to or larger than a predetermined angle is not detected, but the marker detection process is not limited to this.

In S609, the object identification unit 3405 identifies the position of the marker 210 detected in S608 on the 3D model generated in S603. FIG. 13 is an image diagram showing the positions of the markers 210 detected from the captured image of the photographing device 100-1 and the photographed image of the photographing device 100-2 on the 3D model of the vehicle 230. In FIG. 13, the photographing surface 1301 of the photographing apparatus 100-1 and the photographing surface 1302 of the photographing apparatus 100-2 are simulated. The positions of the four markers 210 are (Xn1, Yn1, Zn1), (Xn2, Yn2, Zn2), (Xn3, Yn3, Zn3), respectively, as coordinate information in the coordinate system 120 with the reference position of the race field 130 as the origin. , And (Xn4, Yn4, Zn4). The coordinate information of such a marker 210 indicates the position of the vehicle 230 in the race field 130. As described above, the image generation device 340 may generate a virtual viewpoint image by a method that does not use a 3D model. In this case, the object identification unit 3405 may directly calculate the coordinate information of the marker 210 in the three-dimensional space by using the detection result of the marker 210 from the captured image and the calibration information of the imaging system 110.

In S610, the object identification unit 3405 compares the combination of the detected markers 210 and the coordinate information specified in S609 with the vehicle identification table 700 to identify the vehicle ID of the vehicle to which the detected marker 210 is assigned. do. By identifying the vehicle ID in this way, for example, even when a plurality of vehicles are running in the race track 130, the coordinate information of the marker 210 attached to the vehicle corresponding to the specific vehicle ID is acquired. can. However, if there is only one vehicle with the marker 210 in the race track 130, the process of S610 may be omitted.

In S611, the coordinate conversion unit 3406 specifies the relationship between the coordinate system 120 based on the race field 130 and the coordinate system 220 based on the vehicle 230 based on the coordinate information of the marker 210, and the in-vehicle photographing apparatus 200 Coordinate conversion is performed on the calibration information. FIG. 14 is an image diagram showing the relationship between the coordinate system 120 and the coordinate system 220. The orthogonal coordinates shown by the solid line are the coordinate system 120 with respect to the race field 130, and the Cartesian coordinates shown by the dotted line are the coordinate system 220 with respect to the vehicle 230 integrated inside the coordinate system 120. For example, the origin of the coordinate system 220 is represented by the coordinates (Xg, Yg, Zg) in the coordinate system 120. Further, the coordinate axes of the coordinate system 220 are represented by the directions X', Y', and Z'in the coordinate system 120. Based on such a relationship between the coordinate systems, the coordinate conversion unit 3406 converts the calibration information of the vehicle-mounted photographing apparatus 200 from the coordinates in the coordinate system 220 to the coordinates in the coordinate system 120. For example, the coordinates (Xs1, Ys1, Zs1) indicating the position of the vehicle-mounted photographing apparatus 200 in the coordinate system 220 are converted into the coordinates (Xs2, Ys2, Zs2) in the coordinate system 220. Thereby, the position of the in-vehicle photographing apparatus 200 in the photographing area is specified. In S612, the viewpoint input unit 3407 outputs the calibration information converted in S611 to the viewpoint operation UI 350.

[Viewpoint operation UI operation flow]
15 and 16 are flowcharts showing an example of processing related to determination of a virtual viewpoint and an image switching instruction by the viewpoint operation UI 350. The processes shown in FIGS. 15 and 16 are realized by the CPU 3510 of the viewpoint operation UI 350 expanding the program stored in the ROM 3511 to the RAM 3512 and executing the program. It should be noted that at least a part of the processing shown in FIGS. 15 and 16 may be realized by one or a plurality of dedicated hardware different from the CPU 3510. The same applies to the processing of the flowcharts shown in FIGS. 18 and 19, which will be described later. The processes shown in FIGS. 15 and 16 are executed at the timing when the virtual viewpoint image generation process by the image generation device 340 described with reference to FIG. 12 and the output of the image content by the output device 360 are started. However, the start timing of the processes of FIGS. 15 and 16 is not limited to this. The process of FIG. 15 is executed when the output device 360 outputs the captured image of the vehicle-mounted photographing device 200. On the other hand, when the output device 360 outputs a virtual viewpoint image, the process of FIG. 16 is executed.

First, the process of FIG. 15 will be described. In S701, the viewpoint operation UI 350 sets the mode related to the control of the virtual viewpoint to the viewpoint tracking mode. The viewpoint tracking mode is a mode in which the position of the virtual viewpoint is controlled so as to be a position corresponding to the vehicle-mounted image pickup device 200 (for example, a position corresponding to the vehicle-mounted image pickup device 200). In S702, the viewpoint operation UI 350 acquires the calibration information of the in-vehicle photographing apparatus 200 output in S612 of FIG. 10 from the image generation device 340, and stores the calibration information in the viewpoint storage unit 3505. In S703, the viewpoint operation UI 350 generates viewpoint information based on the calibration information of the vehicle-mounted photographing device 200 so that the position of the virtual viewpoint is the position corresponding to the vehicle-mounted photographing device 200, and the generated viewpoint information is used as an image. Output to the generator 340. In the image generation device 340, a virtual viewpoint image is generated based on this viewpoint information.

In S704, the viewpoint operation UI 350 determines whether or not to switch the image output from the output device 360. For example, when an instruction for image switching is input from the user, it is determined that the image should be switched, and the process proceeds to S705. On the other hand, if the user does not input an image switching instruction, it is determined that the image should not be switched, and the process returns to S702. When the image is not switched, the processes of S702 and S703 are executed every cycle V. The criteria for determining image switching are not limited to the above.

In S705, the image switching instruction unit 3504 instructs the output device 360 to switch the output image from the captured image of the vehicle-mounted photographing device 200 to the virtual viewpoint image. At this point, the virtual viewpoint image generated by the image generation device 340 is a virtual viewpoint image corresponding to the virtual viewpoint at the position corresponding to the in-vehicle photographing device 200. Therefore, even if the image output from the output device 360 is switched from the image captured by the in-vehicle imaging device 200 to the virtual viewpoint image, the composition of the image does not change significantly, and seamless image switching can be realized. In S706, the viewpoint operation UI 350 sets the mode related to the control of the virtual viewpoint to the viewpoint operation mode. The viewpoint operation mode is a mode in which the virtual viewpoint is controlled to change based on the user operation.

In the present embodiment, the position of the virtual viewpoint at a plurality of consecutive time points is determined by the viewpoint operation UI 350 so that the virtual viewpoint moves with the movement of the in-vehicle photographing device 200 while the user operation is not performed in the viewpoint operation mode. It shall be decided. Then, it is assumed that the position of the virtual viewpoint is continuously changed from the position of the in-vehicle photographing apparatus 200 by the user operation. As a result, the viewer does not lose sight of the vehicle 230 after the output image is switched to the virtual viewpoint image. However, the present invention is not limited to this, and the virtual viewpoint may be controlled so as not to move while the user operation is not performed in the viewpoint operation mode.

Next, the process of FIG. 16 will be described. In S801, the viewpoint operation UI 350 accepts a mode switching operation by the user and sets the mode related to the control of the virtual viewpoint to the viewpoint tracking mode. In S802, the viewpoint operation UI 350 acquires the calibration information of the in-vehicle photographing apparatus 200 output in S612 of FIG. 10 from the image generation device 340, and stores the calibration information in the viewpoint storage unit 3505.

In S803, the tracking unit 3503 updates the viewpoint information based on the viewpoint information indicating the current virtual viewpoint and the calibration information of the vehicle-mounted photographing device 200 so that the position of the virtual viewpoint approaches the position of the vehicle-mounted photographing device 200. .. In S804, the viewpoint operation UI 350 outputs the updated viewpoint information to the image generation device 340. In the image generation device 340, a virtual viewpoint image is generated based on this viewpoint information. In the processing example shown in FIG. 16, the viewpoint information is updated so that the position of the virtual viewpoint continuously changes from the original position (for example, the position designated according to the operation by the user) to the position of the in-vehicle photographing apparatus 200. Will be done. Details of this update process will be described later with reference to FIGS. 17 and 18.

In S805, the viewpoint operation UI 350 determines whether or not to switch the image output from the output device 360. For example, when the distance between the position of the virtual viewpoint and the position of the in-vehicle photographing device 200 is less than a predetermined value, it is determined that the image should be switched, and the process proceeds to S806. On the other hand, it is determined that the distance between the position of the virtual viewpoint and the position of the in-vehicle photographing apparatus 200 is equal to or greater than a predetermined value, and the image should not be switched, and the process returns to S802. When the image is not switched, the processes S802 to S804 are executed every cycle V. The criteria for determining image switching are not limited to the above. For example, the viewpoint operation UI 350 may notify the user that the distance between the position of the virtual viewpoint and the position of the vehicle-mounted photographing device 200 is less than a predetermined value. Then, the viewpoint operation UI 350 may determine that the image should be switched in response to the user who received the notification inputting the image switching instruction.

In S806, the image switching instruction unit 3504 instructs the output device 360 to switch the output image from the virtual viewpoint image to the captured image of the in-vehicle imaging device 200. At this point, the distance between the position of the virtual viewpoint corresponding to the virtual viewpoint image generated by the image generation device 340 and the position of the in-vehicle photographing device 200 is less than a predetermined value. Therefore, even if the image output from the output device 360 is switched from the image captured by the in-vehicle imaging device 200 to the virtual viewpoint image, the composition of the image does not change significantly, and seamless image switching can be realized. In S807, the viewpoint operation UI 350 sets the mode related to the control of the virtual viewpoint to the viewpoint operation mode.

Here, with reference to FIG. 17, the control for bringing the position of the virtual viewpoint closer to the position of the in-vehicle photographing apparatus 200 will be described. The vehicle 230 equipped with the in-vehicle photographing device 200 is moving at high speed, and the virtual viewpoint designated by the user operation is not always near the vehicle 230. Therefore, the viewpoint operation UI 350 gradually shortens the distance between the virtual viewpoint and the in-vehicle shooting device 200 over 1 to several seconds, and even if the output image is switched from the virtual viewpoint image to the shot image, the image is switched at a timing with little discomfort. Instruct. The tracking unit 3503 calculates the viewpoint information in each frame so that the virtual viewpoint gradually moves in this way.

FIG. 17 shows an example of the time change of the distance between the vehicle-mounted photographing device 200 and the virtual viewpoint according to the control of the virtual viewpoint before switching the output image from the virtual viewpoint image to the captured image of the vehicle-mounted photographing device 200. The horizontal axis represents time, and the vertical axis represents the distance between the in-vehicle imaging device 200 and the virtual viewpoint. The initial distance between the in-vehicle shooting device 200 and the virtual viewpoint at the time of switching from the viewpoint operation mode to the viewpoint tracking mode is set to "a", and the distance between the virtual viewpoint and the in-vehicle shooting device 200 so as to reduce discomfort at the time of switching is set. Let it be "b". Further, the period required for the distance between the virtual viewpoint and the in-vehicle photographing apparatus 200 to become "b" is defined as "c". The image switching instruction unit 3504 gives a switching instruction when the distance between the virtual viewpoint and the in-vehicle photographing device 200 is reduced to "b", that is, when the period "c" has elapsed from the mode switching instruction. “B” and “c”, which are the criteria for switching determination, are preset and stored in the viewpoint storage unit 3505. After calculating the initial distance "a", the tracking unit 3503 updates the virtual viewpoint so that the distance is shortened to "b" during the period "c". For example, when the frame rate of the virtual viewpoint image is 60 fps and “c” is 2 seconds, the tracking unit 3503 updates the position of the virtual viewpoint 60 × 2 = 120 times to reach the distance “b”. Calculate the amount of change in the virtual viewpoint in one update.

FIG. 18 shows a detailed flow of processing of the tracking unit 3503 in S803 of FIG. In S901, the tracking unit 3503 uses the calibration information of the vehicle-mounted image pickup device 200 acquired in S802 and the current viewpoint information stored in the viewpoint storage unit 3505 to determine the distance between the current vehicle-mounted image pickup device 200 and the virtual viewpoint. To calculate. In S902, the tracking unit 3503 determines whether or not the processing of S803 is the first time in the flow shown in FIG. In the case of the first processing, the process proceeds to S903, and in the case of the second and subsequent processing, the process proceeds to S904. In S903, the tracking unit 3503 calculates how many times the virtual viewpoint is updated in order to bring the virtual viewpoint closer to the in-vehicle photographing device 200.

In S904, the tracking unit 3503 determines the next virtual viewpoint from the distance calculated in S901 and the update count stored in the viewpoint storage unit 3505. Here, the update count is information indicating how many more times the virtual viewpoint is updated before the output image is switched. The update count starts from the number of updates calculated in S903, and is decremented each time the virtual viewpoint is updated. In S905, the tracking unit 3503 reduces the update count by 1 and stores it in the viewpoint storage unit 3505.

As described with reference to FIGS. 17 and 18, by gradually moving the position of the virtual viewpoint closer to the position of the in-vehicle photographing apparatus 200, it is possible to suppress abrupt changes in the virtual viewpoint image output from the output device 360. However, it is possible to provide image content that is easy for the user to see. However, the present invention is not limited to this example, and the viewpoint operation UI 350 may instantaneously move the position of the virtual viewpoint to the position of the in-vehicle photographing device 200 according to the operation of mode switching by the user.

The configuration and operation of the image processing system have been described above. In the above description, a method of realizing seamless image switching by controlling the position of the virtual viewpoint to be a position corresponding to the in-vehicle photographing apparatus 200 and then switching the output image has been described. On the other hand, by controlling the line-of-sight direction from the virtual viewpoint to be the direction corresponding to the vehicle-mounted image pickup device 200 (for example, the direction corresponding to the optical axis direction of the vehicle-mounted image pickup device 200), the output image is switched. It is possible to realize seamless image switching with less discomfort to the viewer. Further, parameters related to the virtual viewpoint image or parameters related to shooting of the vehicle-mounted photographing device 200 so that at least one of the elements such as the angle of view, brightness, and color of the virtual viewpoint image and the captured image of the vehicle-mounted photographing device 200 are brought closer to each other. You may switch the output image after controlling. This makes it possible to further reduce the discomfort of the viewer.

[Modification example]
Here, an example of setting parameters related to the virtual viewpoint image according to the state of the in-vehicle photographing device 200 and the situation of the race field 130 which is the photographing area will be described. In this modification, the viewpoint operation UI 350 executes the process shown in FIG. 19 instead of the process shown in FIG. Other processing contents and the configuration of each device are the same as those in the above-described embodiment. However, the viewpoint operation UI 350 in this modification has a function of acquiring information on the focal length of the in-vehicle photographing apparatus 200 and information on brightness such as ISO and IRIS. For example, the user may input information on the focal length and brightness of the vehicle-mounted photographing device 200 into the viewpoint operation UI 350. Further, the vehicle-mounted photographing apparatus 200 may transmit such information to the viewpoint operation UI 350.

The processing of FIG. 19 will be described below. In S1001, the viewpoint operation UI 350 sets the mode related to the control of the virtual viewpoint to the viewpoint tracking mode. When there are a plurality of vehicles in the race field 130, which is the shooting area, the viewpoint operation UI 350 accepts the user operation for selecting the vehicle to be followed, and executes the migration process with the selected vehicle as the object to be followed. do.

In S1002, the viewpoint operation UI 350 acquires the calibration information of the in-vehicle photographing device 200 from the image generation device 340, and stores the calibration information in the viewpoint storage unit 3505. The calibration information acquired in S1002 includes not only the calibration information of the in-vehicle photographing apparatus 200 installed in the selected vehicle but also the calibration information of the in-vehicle photographing apparatus 200 installed in other vehicles. Is done. However, it is not necessary to acquire the calibration information for all the vehicles in the race track 130. For example, the calibration information for the selected vehicle, the vehicle located in front of the selected vehicle, and the vehicles located in the vicinity thereof may be acquired. You may do it.

In S1003, the viewpoint operation UI 350 identifies the vibration state of the vehicle-mounted photographing apparatus 200 by comparing the current calibration information and the past calibration information regarding the selected vehicle. In S1004, the viewpoint operation UI 350 determines whether or not the in-vehicle photographing device 200 has a vibration of a predetermined magnitude or more. For example, when the position of the vehicle-mounted photographing apparatus 200 in the vertical direction (Z-axis direction) fluctuates by a predetermined width or more, it is determined that there is vibration. When it is determined that there is vibration, in S1005, the viewpoint operation UI 350 changes the parameter related to the virtual viewpoint so that the angle of view of the virtual viewpoint is wider than the angle of view of the in-vehicle photographing apparatus 200 by a predetermined value. As a result, the virtual viewpoint is set so that the shooting range of the vibrating in-vehicle shooting device 200 is included in the field of view according to the virtual viewpoint, and an easy-to-see virtual viewpoint image can be generated.

In S1006, the viewpoint operation UI 350 identifies the state in front of the selected vehicle based on the calibration information about the plurality of vehicles. In S1007, the viewpoint operation UI 350 determines whether another vehicle is present in front of the selected vehicle. When it is determined that another vehicle exists, in S1008, the viewpoint operation UI 350 changes the position of the virtual viewpoint upward by a predetermined value from the position of the in-vehicle photographing device 200. As a result, the virtual viewpoint is set so that the range shielded by the vehicle in front of the selected vehicle in the field of view according to the virtual viewpoint becomes small, and a virtual viewpoint image that makes it easy to confirm the front of the selected vehicle is generated. become able to. That is, the position of the virtual viewpoint set as the position corresponding to the position of the in-vehicle photographing device 200 may be the position corresponding to the position of the in-vehicle photographing device 200 depending on the situation in the race field 130 which is the photographing area. If there is, it will be a position separated from the position of the in-vehicle photographing apparatus 200 by a predetermined distance.

In S1009, the viewpoint operation UI 350 specifies a parameter related to shooting of the vehicle-mounted photographing device 200 installed in the selected vehicle. The parameters specified include, for example, parameters related to the focal length and the brightness of the captured image. In S1010, the viewpoint operation UI 350 determines whether there is a difference of a threshold value or more between the parameter related to the virtual viewpoint and the parameter related to the shooting of the vehicle-mounted photographing device 200. When it is determined that there is a difference in the parameters, in S1011, the viewpoint operation UI 350 changes the parameters related to the virtual viewpoint according to the parameters related to the shooting of the in-vehicle shooting device 200.

In S1012, the viewpoint operation UI 350 outputs the viewpoint information indicating the virtual viewpoint reflecting the above change to the image generation device 340. In the image generation device 340, a virtual viewpoint image is generated based on this viewpoint information. In S1014, the viewpoint operation UI 350 determines whether or not to switch the image output from the output device 360. For example, when an instruction for image switching is input from the user, it is determined that the image should be switched, and the process proceeds to S1014. On the other hand, if the user does not input an image switching instruction, it is determined that the image should not be switched, and the process returns to S1002. When the image is not switched, the processes of S1002 to S1012 are executed every cycle V. The criteria for determining image switching are not limited to the above. In S1015, the image switching instruction unit 3504 instructs the output device 360 to switch the output image from the captured image of the vehicle-mounted photographing device 200 to the virtual viewpoint image. In S706, the viewpoint operation UI 350 sets the mode related to the control of the virtual viewpoint to the viewpoint operation mode.

According to the above processing shown in FIG. 19, even when the brightness of the image captured by the in-vehicle photographing apparatus 200 is changed due to a change in external light or the like, or when the focal length of the in-vehicle photographing apparatus 200 is changed, a sense of incongruity is felt. Small seamless image switching can be realized. Further, when the captured image of the in-vehicle photographing device 200 is difficult to see due to image blurring due to vibration, or when other vehicles are crowded in front of the vehicle of interest and it is difficult to understand the situation, the viewer takes a picture of the in-vehicle photographing device 200. You can watch while switching from the image to the easy-to-see virtual viewpoint image as appropriate. This makes it possible to provide the user with a more satisfying viewing experience.

As described above, in the viewpoint operation UI 350 in the present embodiment, the position of the in-vehicle imaging device 200 that moves within the imaging region to be captured by the plurality of imaging devices 100 from different positions is obtained by data based on the imaging by the imaging device 100. Is specified using. Further, the viewpoint operation UI 350 sets the position of the virtual viewpoint corresponding to the virtual viewpoint image generated based on the plurality of images acquired by the plurality of shooting devices 100 to the position corresponding to the position of the specified in-vehicle shooting device 200. decide. Then, the virtual viewpoint image corresponding to the virtual viewpoint determined in this way is generated by the image generation device 340, and the image content including the virtual viewpoint image and the captured image acquired by the in-vehicle photographing device 200 is output by the output device 360. Will be done. According to the above configuration, in the image content including the captured image and the virtual viewpoint image acquired by the in-vehicle imaging device 200, the relationship between the viewpoints corresponding to the captured image and the virtual viewpoint image can be easily recognized. It becomes. As a result, it is possible to provide image content with high user satisfaction.

The shape of the vehicle basically does not change during the race except for the tires. Therefore, if the positional relationship between the marker 210 attached to the vehicle and the in-vehicle photographing device 200 is specified in advance, the position of the in-vehicle photographing device 200 can be specified to some extent accurately by detecting the marker 210 during the race. In the present embodiment, a method of providing image content with high viewer satisfaction by utilizing such a feature has been described. However, the method for specifying the position of the in-vehicle photographing device 200 is not limited to this. For example, the position of the vehicle-mounted photographing device 200 may be specified by detecting a marker attached to the vehicle-mounted photographing apparatus 200 or a feature point of the vehicle-mounted photographing apparatus 200 from the captured image acquired by the photographing system 110.

The object on which the in-vehicle photographing device 200 is installed is not limited to the above-mentioned four-wheeled vehicle, and may be, for example, a two-wheeled vehicle. Further, the target of shooting by the shooting system 110 is not limited to motor sports, and may be other competitions or events. For example, the photographing system 110 may photograph a field in which a game such as soccer or rugby is played, and the image generation device 340 may generate a virtual viewpoint image based on the photographed image obtained by the photographing system 110. Then, the output device 360 may output image content including a photographed image and a virtual viewpoint image of the photographing device installed in the drone moving over the field. In this case, the viewpoint operation UI 350 may determine the position of the virtual viewpoint corresponding to the virtual viewpoint image according to the position of the photographing device installed in the drone.

Further, for example, when image content including a photographed image of a photographing device attached to a person or an animal and a virtual viewpoint image is generated, the viewpoint operation UI 350 sets the position of the virtual viewpoint corresponding to the virtual viewpoint image to the person or animal. It may be determined according to the position of the photographing apparatus mounted on the. The position of such a photographing device may be specified based on information from a position sensor such as GPS possessed by the photographing device, or may be acquired by another photographing device that photographs an area in which a person or an animal moves. It may be specified based on the result of detecting a person or an animal from the captured image. Even with such a configuration, it is possible to provide image content in which the relationship between the viewpoints of the photographed image and the virtual viewpoint image can be easily recognized, as in the above-described embodiment.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a recording medium readable by a computer.

110 Imaging system 200 In-vehicle imaging device 320 Control device 340 Image generator 360 Output device

Claims (23)

The position of the mobile photographing device for acquiring the captured image included in the image content to be generated and moving within the photographing area to be photographed from different positions by the plurality of photographing devices is set to the plurality of positions. Specific means to identify using the data based on the shooting by the shooting device included in the shooting device,
The position of the virtual viewpoint corresponding to the virtual viewpoint image included in the image content and generated based on the plurality of images acquired by the plurality of photographing devices is specified by the specific means. An information processing apparatus comprising: a determination means for determining a position according to a position of a mobile photographing apparatus. The information processing device according to claim 1, wherein the mobile photographing device is a camera installed on an object moving in the photographing area. The specific means uses information indicating a relative position of the moving photographing device with respect to a reference position in the object and data indicating the position of the object acquired as data based on the photographing, and the position of the moving photographing device. The information processing apparatus according to claim 2, wherein the information processing apparatus is specified. The information indicating the relative position of the mobile photographing device is information indicating the coordinates of the position of the moving photographing device in the first coordinate system having the reference position in the object as the origin.
The data indicating the position of the object is data indicating the coordinates of the position of the object in the second coordinate system having the reference position in the photographing region as the origin.
The specific means is characterized in that the position of the mobile photographing device is specified by converting the coordinates of the position of the moving photographing device in the first coordinate system according to the second coordinate system. Item 3. The information processing apparatus according to item 3. The information processing apparatus according to claim 2, wherein the specifying means specifies the position of the mobile photographing apparatus by using the data representing the three-dimensional shape of the object acquired as the data based on the photographing. .. It has an identification means for identifying a specific object among a plurality of objects moving in the photographing area by using data based on photography by the photographing apparatus included in the plurality of photographing devices.
The information processing apparatus according to any one of claims 2 to 5, wherein the specific means specifies the position of the mobile photographing device installed on the specific object identified by the identification means. .. The information processing apparatus according to any one of claims 2 to 6, wherein the object moving in the photographing area is a vehicle or a drone. The information processing apparatus according to any one of claims 1 to 7, wherein the determination means determines the position of the virtual viewpoint at a position corresponding to the position of the moving photographing device. The information processing according to any one of claims 1 to 7, wherein the determination means determines the position of the virtual viewpoint with respect to the position of the mobile photographing apparatus according to a situation in the photographing area. Device. Any of claims 1 to 9, wherein the determination means determines the position of the virtual viewpoint at a plurality of consecutive time points so that the virtual viewpoint moves with the movement of the moving photographing device. The information processing apparatus according to item 1. Claims 1 to 10 include a control means for controlling the position of the virtual viewpoint so as to continuously change from a position designated according to an operation by a user to a position determined by the determination means. The information processing apparatus according to any one of the following items. The invention according to any one of claims 1 to 10, further comprising a control means for controlling the position of the virtual viewpoint so as to continuously change from the position determined by the determination means according to a user operation. Information processing equipment. Claims 1 to 1, characterized in that, in the image content, a virtual viewpoint image corresponding to a virtual viewpoint at a position determined by the determination means and a photographed image acquired by the moving photographing device are switched and displayed. The information processing apparatus according to any one of 12. Claims 1 to 12 are characterized in that, in the image content, a virtual viewpoint image corresponding to a virtual viewpoint at a position determined by the determination means and a photographed image acquired by the moving photographing device are simultaneously displayed. The information processing apparatus according to any one of the above items. The specifying means further identifies the optical axis direction of the mobile photographing device by using data based on photography by the photographing devices included in the plurality of photographing devices.
The determination means is further characterized in that the line-of-sight direction from the virtual viewpoint corresponding to the virtual viewpoint image is determined in a direction corresponding to the optical axis direction of the moving photographing device specified by the specific means. Item 6. The information processing apparatus according to any one of Items 1 to 14. Any of claims 1 to 15, further comprising a setting means for setting parameters relating to at least one of the angle of view and brightness corresponding to the virtual viewpoint image based on the parameters related to shooting by the moving shooting device. The information processing device according to item 1. The information processing apparatus according to any one of claims 1 to 15, further comprising a setting means for setting parameters related to an angle of view corresponding to the virtual viewpoint image based on a vibration state of the moving photographing device. .. The claim is characterized by having a generation means for generating the virtual viewpoint image corresponding to the position of the virtual viewpoint determined by the determination means based on the plurality of images acquired by the plurality of photographing devices. The information processing apparatus according to any one of 1 to 17. 18. The invention according to claim 18, further comprising an output means for outputting the image content generated by using the photographed image acquired by the mobile photographing apparatus and the virtual viewpoint image acquired by the generating means. Information processing equipment. The position of the mobile photographing device for acquiring the captured image included in the image content to be generated and moving within the photographing area to be photographed from different positions by the plurality of photographing devices is set to the plurality of positions. A specific process to specify using data based on shooting by the shooting device included in the shooting device, and
The position of the virtual viewpoint corresponding to the virtual viewpoint image included in the image content and generated based on the plurality of images acquired by the plurality of photographing devices is specified in the specific step. An information processing method comprising: a determination step of determining a position according to a position of a mobile photographing apparatus. The information processing method according to claim 20, wherein the mobile photographing device is a camera installed on an object moving in the photographing area. In the specific step, the information indicating the relative position of the mobile photographing device with respect to the reference position in the vehicle and the data indicating the position of the vehicle acquired as the data based on the photographing are used to obtain the moving photographing device. The information processing method according to claim 21, wherein the position is specified. A program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 19.

Images