JP2022032491A - Video processing apparatus, video processing method, and program - Google Patents

Abstract

To reduce viewer discomfort at switching between a virtual viewpoint image and a picked-up image.SOLUTION: An object size calculation unit calculates size of a reference object for a frame in an actual camera video, the frame immediately before a pan angle of an actual camera video imaging apparatus exceeds a predetermined threshold acquired as switching reference information. The object size calculation unit calculates a virtual camera parameter in which the size of the reference object in a virtual viewpoint video becomes substantially the same as the size of the reference object in the actual camera video. A virtual camera path creation unit creates a virtual camera path in which the calculated virtual camera parameter is set at a starting position of a virtual camera. A virtual viewpoint video creation unit creates a virtual viewpoint video on the basis of the created virtual camera path and outputs the video to a video switching unit. When the video switching unit acquires the virtual viewpoint video, the video switching unit outputs the virtual viewpoint video subsequent to the frame in the actual camera video for which the size of the reference object is calculated.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for switching between a captured image and a virtual viewpoint image.

The same subject (object) is synchronously imaged by multiple image pickup devices installed at different positions, and the obtained multi-viewpoint image is used to virtually arrange an image pickup device (virtual viewpoint) that does not actually exist in a three-dimensional space. ) Is attracting attention as a technology for generating virtual viewpoint images.

This virtual viewpoint image can also be combined with an image captured image (hereinafter, simply referred to as an image captured image) generated by one image pickup device such as a broadcast image pickup device. For example, in a soccer game, a series of subjective cuts with a virtual viewpoint image, such as the ball passing by the keeper's eyes in a shooting scene, and cuts with an captured image that captures the detailed facial expressions of the keeper and kicker with a high-quality zoom are connected. You can create a goal scene. In this way, by adding a virtual viewpoint image such as a subjective viewpoint for each player, which cannot be taken by a normal image pickup device, to the high-definition image, it can be expected to provide the viewer with a more realistic image experience. ..

In order to switch between the virtual viewpoint image and the captured image with less discomfort, Patent Document 1 discloses that the virtual viewpoint image is automatically controlled so that the composition and image quality before and after the switching between the virtual viewpoint image and the captured image are the same. ing.

Japanese Unexamined Patent Publication No. 2020-42665

However, in Patent Document 1, even if the composition at the time of switching between the virtual viewpoint image and the captured image is the same, the object size is not always the same, so that a difference occurs between the frames of the virtual viewpoint image and the captured image, and the viewer at the time of switching. However, there is a problem that it may feel uncomfortable.

An object of the present disclosure technique is to reduce the discomfort of the viewer when switching between the virtual viewpoint image and the captured image.

The technique of the present disclosure corresponds to a parameter acquisition means for acquiring an image pickup parameter related to an image pickup device that images a predetermined object, and the predetermined object in the image pickup image obtained by the image pickup performed by the image pickup device. A virtual viewpoint image based on the size acquisition means for acquiring the size of the area to be measured, the imaging parameters acquired by the parameter acquisition means, and the size of the area corresponding to the predetermined object acquired by the size acquisition means. A path generation means for generating a virtual viewpoint path representing a time change of the virtual viewpoint according to the above, a virtual viewpoint path generated by the path generation means, and an image obtained by photographing the predetermined object from a plurality of directions. Based on the above, it is characterized by having an image generation means for generating a virtual viewpoint image and an output means for selecting and outputting one of the captured image and the virtual viewpoint image generated by the image generation means. do.

According to the technique of the present disclosure, it is possible to reduce the discomfort of the viewer when switching between the virtual viewpoint image and the captured image.

FIG. 3 is an overall configuration diagram of a video generation system according to the first embodiment. It is a flow diagram of the switching process to the virtual viewpoint image in Embodiment 1. FIG. It is a figure explaining the image switching position from the real camera to the virtual camera in Embodiment 1. FIG. It is a flow diagram of the object size adjustment process at the time of switching to a virtual viewpoint image in Embodiment 1. FIG. It is an overall block diagram of the image generation system in Embodiment 2. FIG. It is a flow diagram of the switching process to the virtual viewpoint image in Embodiment 2. It is a flow diagram of the object size adjustment process at the time of switching to a virtual viewpoint image in Embodiment 2. FIG. It is a figure explaining the example of how to adjust the object size in Embodiment 2. It is a figure explaining the camera path mode in Embodiment 2. FIG. It is a hardware block diagram of the virtual viewpoint image generation apparatus in Embodiments 1 and 2.

Hereinafter, embodiments will be described with reference to the drawings. The configuration shown in the following embodiments is only an example, and the technique of the present disclosure is not limited to the illustrated configuration.

(Embodiment 1)
FIG. 1 is an overall configuration diagram of a video processing system according to the first embodiment. The image processing system includes a virtual viewpoint image generation device 10, a virtual viewpoint image imaging device group 104, an actual camera image imaging device 106, an actual camera imaging area setting unit 107, and an image switching device 108.

The virtual viewpoint image in the present embodiment is composed of an image corresponding to a virtual viewpoint specified by a user based on a multi-view image obtained by capturing the same area from a plurality of directions by a plurality of image pickup devices, and is a free viewpoint image. It is also called. The virtual viewpoint video is not limited to the video corresponding to the viewpoint freely (arbitrarily) specified by the user, and for example, the video corresponding to the viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint video.

The virtual camera in the present embodiment is a virtual camera different from the plurality of image pickup devices actually installed, and is a concept for expediently explaining the virtual viewpoint related to the generation of the virtual viewpoint image. That is, the virtual viewpoint image can be regarded as an image captured from a virtual viewpoint set in the three-dimensional space associated with the imaging region. Then, the position and orientation of the viewpoint in the virtual image pickup can be expressed as the position and orientation of the virtual camera. In other words, it can be said that the virtual viewpoint image is an image simulating the captured image obtained by the camera, assuming that the camera exists at the position of the virtual viewpoint set in the three-dimensional space. Further, in the present embodiment, the trajectory of the transition of this virtual viewpoint is referred to as a “virtual camera path”. The virtual camera path can represent not only the transition of the position of the virtual viewpoint but also the transition of the orientation from the virtual viewpoint. However, it is not essential to use the concept of a virtual camera in order to realize the configuration of this embodiment. That is, at least the information indicating a specific position and the information indicating the direction in the three-dimensional space are set, and the virtual viewpoint image may be generated according to the set information.

(Explanation of configuration)
The virtual camera path operation unit 101 controls the virtual camera based on the instruction input received from the virtual camera operator, and outputs the control result as a virtual camera parameter to the virtual camera path generation unit 102. Here, the virtual camera parameter includes at least a parameter indicating the position and orientation (orientation) of the virtual camera. However, the present invention is not limited to this, and for example, a parameter indicating the angle of view of the virtual camera, specifically, a zoom value and the like may be included.

The virtual camera path generation unit 102 generates a virtual camera path representing a time change of the virtual camera parameter based on the virtual camera parameters output from the virtual camera path operation unit 101 and the object size calculation unit 105. Here, the virtual camera path is represented by the virtual camera parameters set for each of the temporally continuous frames. When generating a virtual camera path, the virtual camera path generation unit 102 associates the virtual camera parameter with the time code so that which parameter corresponds to which frame can be specified.

The virtual viewpoint image generation unit 103 generates a three-dimensional model from the multi-viewpoint image captured by the virtual viewpoint image imaging device group 104. Then, the virtual viewpoint image is generated by mapping the texture based on the virtual viewpoint (position, posture, angle of view of the virtual camera) in the virtual camera path generated by the virtual camera path generation unit 102. The virtual viewpoint image generation unit 103 outputs the generated virtual viewpoint image to the image switching unit 108. Further, the virtual viewpoint image generation unit 103 generates a virtual viewpoint image based on the virtual viewpoint parameter calculated by the object size calculation unit 105.

The virtual viewpoint image imaging device group 104 performs synchronous imaging by a plurality of real cameras installed so as to surround the competition field and the like, and outputs the captured multi-viewpoint image to the virtual viewpoint image generation unit 103.

The object size calculation unit 105 calculates the size of a reference foreground object (hereinafter referred to as a reference object) in the actual camera image output from the actual camera image imaging device 106, and acquires the size. When there are a plurality of foreground objects in the video, the reference object may be, for example, the in-focus object or a foreground object having a specific appearance feature. Further, any foreground object specified by the user may be used as the reference object. In each frame of the actual camera image, a pixel area corresponding to the reference object is extracted, and for example, the maximum number of pixels in each of the vertical direction and the horizontal direction of the pixel area is calculated as the size of the reference object. The reference object may be, for example, the whole body of a person, a part of the body of a person, or an object such as a ball. Further, the object size calculation unit 105 acquires parameters (hereinafter, referred to as “imaging parameters”) regarding the actual camera image imaging device 106.

The object size calculation unit 105 generates a virtual camera parameter in which the size of the reference object in the virtual viewpoint image is substantially equal to the size in the actual camera image based on the calculated size of the reference object and the acquired imaging parameter. The object size calculation unit 105 outputs the generated virtual camera parameters to the virtual camera path generation unit 102. The imaging parameters include at least information indicating the position and orientation (orientation) of the actual camera. However, the imaging parameter is not limited to this, and may include information indicating the angle of view of the actual camera. The method of calculating the size of the object in the video and the process of adjusting the size will be described later. In the virtual camera parameters generated based on the imaging parameters, not only the size of the reference object is almost the same as that of the actual camera image, but also the composition of the image is the same, so the difference in the image at the time of switching becomes small. It is possible to reduce the discomfort of the viewer.

The actual camera image imaging device 106 is a broadcasting camera, a drone camera, a spider camera, or the like, and the actual camera image is an image captured by the actual camera image imaging device 106. Further, the actual camera image imaging device 106 outputs operation information (pan angle, tilt angle, zoom value, movement amount) to the actual camera imaging area setting unit 107.

The actual camera image pickup area setting unit 107 presets an operation range in which the actual camera image pickup device 106 can image an object with a predetermined size in the vicinity of the image switching position based on user input. Specifically, in the pan operation of the actual camera image imaging device 106, an operation range is set with the upper and lower limit values of the pan angle that can be followed by a predetermined size when the object is imaged at the maximum telephoto. The area in which the real camera image image pickup device 106 can take an image within this operation range is an image pickup area in which the real camera image image pickup device 106 can take an image of the reference object in a predetermined size by adjusting the zoom value.

Here, the pan angle is shown as an example of the operation information for setting the operation range, but it may be a movement amount, a zoom value, or a tilt angle. Then, the actual camera image pickup area setting unit 107 acquires the operation information of the actual camera image image pickup device 106 and compares it with the threshold value set for each operation, so that the actual camera image image pickup device 106 exceeds the image switching position. Judge whether or not. The actual camera image pickup area setting unit 107 outputs the determination result to the image switching unit 108.

The image switching unit 108 selects either the virtual viewpoint image output from the virtual viewpoint image generation unit 103 or the actual camera image output from the actual camera image imaging device 106, and broadcast equipment (not shown). And output to the distribution server. The image switching unit 108 selects and switches the image based on the determination result output by the actual camera image pickup area setting unit 107 (whether or not the operation information of the actual camera image image pickup device 106 exceeds the set operation range). ..

In the above-described configuration, the image pickup device included in the virtual viewpoint image image pickup device group 104 and the actual camera image pickup device 106 are described as different from each other, but they may be the same. That is, the image of the image pickup device included in the image pickup device group 104 for the virtual viewpoint image and the virtual viewpoint image may be switched.

FIG. 10 shows a hardware configuration of an information processing device 200 that can operate as the virtual viewpoint video generation device 10 of FIG. The information processing device 200 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication I / F (interface) 217, and a bus 218.

The CPU 211 executes a computer program stored in the ROM 212 or the RAM 213 and processes the data stored in the computer program. The information processing apparatus 200 may have one or more dedicated hardware different from the CPU 211, and the dedicated hardware may execute at least a part of the processing by the CPU 211. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 212 stores programs and the like that do not require changes. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, data supplied from the outside via the communication I / F 217, and the like. The auxiliary storage device 214 is composed of, for example, an HDD (hard disk drive) or the like, and stores various data such as image data and audio data.

The display unit 215 is composed of, for example, a liquid crystal display or the like, and displays a GUI (graphical user interface) for the user to operate the information processing apparatus 200. The operation unit 216 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and receives an operation by a user and inputs various instructions to the CPU 211. The communication I / F 217 is used for communication with an external device of the information processing device 200. For example, when the information processing device 200 is connected to an external device by wire, a communication cable is connected to the communication I / F 217. When the information processing device 200 has a function of wirelessly communicating with the external device 200, the communication I / F 217 includes an antenna. The bus 218 connects each part of the information processing apparatus 200 to transmit information.

In the present embodiment, the display unit 215 and the operation unit 216 will be described as existing inside the information processing device 200, but at least one of the display unit 215 and the operation unit 216 is another device outside the information processing device 200. May exist as. In this case, the CPU 211 may operate as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216.

(Explanation of flowchart)
FIG. 2 is a flowchart illustrating a process of switching from a real camera image to a virtual viewpoint image according to the present embodiment.

In S201, the image switching unit 108 acquires switching reference information for determining whether or not the imaging area of the actual camera image imaging device 106 exceeds the image switching position from the user input to the actual camera imaging area setting unit 107. Specifically, the switching reference information is a threshold value that defines an operation range of the operation information of the actual camera image capturing apparatus 106. Since this switching reference information defines the boundary of the imaging region of the actual camera image imaging device 106, it is possible to set the position for switching from the actual camera image to the virtual viewpoint image based on the switching reference information.

(Explanation of switching position)
Here, in FIG. 3, the imaging region of the real camera image imaging device 106 and the image switching position to the virtual camera will be described. 301 indicates a real camera image imaging device, and 302 indicates a virtual camera. The broken line of 303 indicates the image switching position between the real camera 301 and the virtual camera 302. 304 indicates a virtual camera path. In the description of FIG. 3, for convenience, the actual camera image imaging device 301 is fixed near the center of the pitch, and the object is chased by the pan operation as an example. However, the object may be chased by the tilt operation, and the implementation method is this. Not limited to. Further, when the installation position of the actual camera image imaging device 301 is not fixed, the movement operation of the actual camera image imaging device 301 may be combined.

Since the area to the right of the image switching position 303 is an area that can be imaged by panning within the operation range set in the actual camera image imaging device 301, when a reference object exists in this area, the image switching unit 108 actually starts. The camera image is output. Further, since the left side from the image switching position 303 is an area that cannot be imaged by panning operation within the operation range set in the actual camera image imaging device 301, when a reference object exists in this area, the image switching unit 108 Virtual viewpoint video is output.

The image switching position 303 is, for example, among the upper and lower limit values of the pan angle that allows the actual camera image imaging device 301 to image the reference object at a predetermined size at the maximum telephoto, which is the operating range of the actual camera image imaging device 301. It can be set based on one.

The size of the reference object in the image when switching from the real camera image to the virtual viewpoint image is determined in advance before imaging, and the image switching position 303 is determined based on the predetermined size. The size of the reference object in the image is determined by the distance from the installation position of the actual camera image imager 106 to the object and the specifications of the actual camera image imager 106 such as the lens, the zoom value, and the size of the image sensor. Therefore, by determining the size of the reference object, the image switching position 303 can be determined from the specifications and the installation position of the actual camera image imaging device 106.

In S202, the image switching unit 108 outputs the image obtained by imaging with the actual camera image imaging device 106. This is a state in which the actual camera image imaging device 106 captures the reference object within the imaging region of the actual camera image imaging device 106, and the operation information of the actual camera image imaging device 106 does not exceed the threshold value set in S201. It is shown that.

In S203, the actual camera image pickup area setting unit 107 determines whether or not the image obtained by imaging with the actual camera image image pickup device 106 exceeds the image switching position 303, and outputs the determination result to the image switching unit 108. do. Whether or not the image exceeds the image switching position 303 is determined based on whether or not the pan angle of the actual camera image device 106 exceeds a predetermined threshold value (angle) acquired as switching reference information. If the image switching position 303 is not exceeded (No in S203), the process proceeds to S202 and the image switching unit 108 continuously outputs the actual camera image. If the video switching position 303 is exceeded (Yes in S203), the process proceeds to S204.

In S204, the object size calculation unit 105 determines the size of the reference object for the frame captured immediately before the pan angle of the actual camera image imaging device 106 exceeds the predetermined threshold value acquired as the switching reference information in the actual camera image. calculate.

In S205, the object size calculation unit 105 calculates a virtual camera parameter in which the size of the reference object in the virtual viewpoint image is substantially the same as the size of the reference object calculated in S204. Detailed processing will be described later with reference to FIG. The object size calculation unit 105 outputs the calculated virtual camera parameters to the virtual camera path generation unit 102.

In S206, the virtual camera path generation unit 102 generates a virtual camera path in which the virtual camera parameter calculated in S205 is set at the start position of the virtual camera. The virtual camera path after the start position is generated according to the input from the virtual camera operator via the virtual camera operation unit 101. The virtual camera path generation unit 102 outputs the generated virtual camera path to the virtual viewpoint image generation unit 103.

In S207, the virtual viewpoint image generation unit 103 generates a virtual viewpoint image based on the virtual camera path acquired from the virtual camera path generation unit 102 in S206, and outputs the virtual viewpoint image to the image switching unit 108.

In S208, when the image switching unit 108 acquires the virtual viewpoint image generated by the virtual viewpoint image generation unit 103 in S207, the image switching unit 108 outputs the virtual viewpoint image following the frame of the actual camera image for which the size of the reference object is calculated in S204. do.

(Explanation of object size)
FIG. 4 is a flowchart illustrating a method of calculating the object size and a method of adjusting the object size. This is the details of the process of S205 in FIG.

First, in S401, the object size calculation unit 105 detects a reference object from the image output from the actual camera image image pickup device 106. As for the method of detecting the reference object, a technique of separating the background and the object may be used, or information may be acquired from the actual camera image capturing apparatus 106 by using a technique such as object recognition.

In S402, the object size calculation unit 105 measures the size of the reference object detected in S401, that is, the size of the image area corresponding to the reference object in the frame of the image output from the actual camera image pickup device 106. In the present embodiment, the number of pixels in the vertical direction is measured, but the measurement method is not limited to this, and other measurement methods such as measuring the number of pixels in the horizontal direction may be used.

In S403, the object size calculation unit 105 acquires the image pickup parameters (operation information, position) of the actual camera image image pickup device 106, and outputs the image pickup parameters to the virtual viewpoint image generation unit 103 as virtual camera parameters.

In S404, the virtual viewpoint image generation unit 103 sets the virtual camera with the virtual camera parameters output in S403, and generates a virtual viewpoint image corresponding to the first frame of the virtual viewpoint image generated in S207. Setting the virtual camera with the virtual camera parameter output in S403 means adjusting the virtual camera to the position and orientation of the actual camera image capturing device 106. The virtual viewpoint image generated here is a frame of the actual camera image used for calculating the size of the reference object in S204 in the image obtained by combining the actual camera image and the virtual viewpoint image output from the image switching unit 108. It is desirable to have adjacent frames. Therefore, for the multi-viewpoint image used here to generate the virtual viewpoint image, use one having a time code corresponding to the next frame of the frame of the actual camera image used for calculating the size of the reference object in S204. Is desirable.

In S405, the object size calculation unit 105 measures the size of the reference object in the virtual viewpoint image generated in S404. Then, the object size calculation unit 105 compares the size of the reference object measured in the virtual viewpoint image generated in S404 with the size of the reference object measured in the actual camera image in S402. Specifically, it is determined whether or not the difference between the size in the real camera image and the size in the virtual viewpoint image is larger than the threshold value. When the difference is equal to or less than the threshold value, it is considered that the sizes of the reference objects are equal, the process of FIG. 4 is terminated, and the process proceeds to the process of S206 or later. If the difference is larger than the threshold value, the process proceeds to S406.

In S406, the object size calculation unit 105 outputs the virtual camera parameters whose zoom value and position are adjusted in order to match the size of the reference object compared in S405 to the virtual viewpoint image generation unit 103, and returns to S404.

In the present embodiment, the case of switching from the real camera image to the virtual viewpoint image has been described, but the present invention can also be applied to the case of switching from the virtual viewpoint image to the real camera image. However, when switching from the virtual viewpoint image to the actual camera image, the virtual viewpoint parameter at the end position of the virtual camera path is set based on the imaging parameter when the image switching position 303 is imaged by the actual camera image imaging device 106. .. Further, in this case, since the virtual viewpoint image is output from the image switching unit 108 before the actual camera image, the actual camera image is not output in S202, but the actual camera image is output after S207. To. In order to enable such processing, it is conceivable that the image switching unit 108 is provided with a storage unit for temporarily holding the actual camera image.

In this way, by matching the sizes of the objects in the real camera image and the virtual viewpoint image at the time of image switching, the viewer can reduce the discomfort.

(Embodiment 2)
In the present embodiment, the position of the virtual camera is first determined according to the value set in the virtual camera path setting unit 509, and then the size of the reference object is adjusted by the zoom value.

This determines the position of the virtual camera based on the position of the real camera when aligning the reference object size in the real camera image and the reference object size in the virtual viewpoint image in the first embodiment. It will impose restrictions on the path.

When you want to create a virtual camera path closer to the object from a position closer to the object, if the initial position of the virtual camera is set to a position far from the object, the virtual camera will move rapidly until the desired virtual camera path is reached. I need to let you. However, if the movement speed of the virtual viewpoint is limited to the extent that the virtual viewpoint image does not become unnatural, the movement may not be in time and a decisive scene may not be reproduced. This is the same problem when you want to create a bird's-eye view virtual camera path from a position far from the object, even if the initial position of the virtual camera is set near the object.

Therefore, in the present embodiment, it is possible to relax the restrictions on the virtual camera path after switching the virtual cameras while minimizing the discomfort when switching the images.

(Explanation of configuration)
FIG. 5 is an overall configuration diagram of the video processing system according to the second embodiment, but the difference from the first embodiment will be mainly described.

The virtual camera path setting unit 509 sets the distance (whether the virtual camera path is far or near from the object, etc.) of the virtual camera path generated after switching the virtual viewpoint image, and outputs the distance to the object size calculation unit 505. .. The virtual camera operator sets the distance between this virtual viewpoint and the reference object. This setting may be set in several stages, and the virtual camera position described later may be changed accordingly.

The object size calculation unit 505 generates a virtual camera path parameter from the image pickup parameter output from the actual camera image image pickup device 506 and the setting of the virtual camera path setting unit 509, as in the first embodiment. The generation of the virtual camera parameter is processed in the same manner as in the first embodiment, except that the following points are different. Since the other operations are the same as those in the first embodiment, the description thereof will be omitted.

(Explanation of flowchart)
FIG. 6 is a flowchart illustrating a process of switching from a real camera image to a virtual viewpoint image.

Since the process of S601 is the same process as S201 of FIG. 2 of the first embodiment, the description thereof will be omitted.

In S602, the virtual viewpoint camera path setting unit 509 sets the virtual camera path mode (whether the distance between the virtual camera and the reference object is close or far) of the virtual viewpoint path to be generated after this according to the input from the virtual camera operator. The virtual camera path mode will be described later.

Since the flow of S603 to S605 is the same process as S202 to S204 of FIG. 2 of the first embodiment, the description thereof will be omitted.

In S606, the object size calculation unit 505 determines the position of the virtual camera according to the set virtual camera path mode, and then adjusts the zoom value to align the size of the actual camera image and the reference object with the virtual camera parameter. To generate. The object size calculation unit 505 outputs the generated virtual camera parameters to the virtual camera path generation unit 502. Detailed processing will be described later.

Since the processes of S607 and S608 are the same as those of S201 of FIG. 2 of the first embodiment, the description thereof will be omitted.

(Virtual camera parameter generation)
FIG. 7 is a flowchart illustrating a method of calculating the object size and a method of adjusting the size. This corresponds to the processing of S606 in FIG.

Since the processes of S701 to S703 are the same as those of S401 to S403 of FIG. 4 of the first embodiment, the description thereof will be omitted.

In S704, the object size calculation unit 505 corrects the parameters related to the position of the virtual camera among the virtual camera parameters output in S703 according to the virtual camera path mode set in the virtual camera path setting unit 509. The object size calculation unit 505 outputs the corrected virtual camera parameter to the virtual viewpoint image generation unit 503.

Since the processes of S705 and S706 are the same as those of S404 and S405 of FIG. 4 of the first embodiment, the description thereof will be omitted.

In S707, the object size calculation unit 505 outputs the virtual camera parameter adjusted by adjusting the zoom value of the virtual camera in order to match the size of the reference object compared in S706 to the virtual viewpoint image generation unit 503, and returns to S705.

(Relationship with camera path)
In FIG. 8, how to adjust the size of the reference object will be described.

801 is the position of the virtual camera assuming that the virtual camera path after switching the virtual viewpoint image is created at a position close to the reference object at a wide angle. By setting the zoom value to the wide-angle side, the position of the virtual camera is brought closer to the object. As a result, the position of the virtual camera can be set to a position close to the reference object, so that it becomes easy to create a virtual camera path at a position close to the reference object.

802 is the position of the virtual camera assuming that the virtual camera path after switching the virtual viewpoint image is created at a position far from the reference object in the telephoto position. This sets the zoom value to the telephoto side, which keeps the position of the virtual camera away from the object. As a result, the position of the virtual camera is set at a position away from the reference object, so that it becomes easy to create a virtual camera path for a bird's-eye view.

(Virtual camera path mode setting)
Further, the present invention is not limited to these two cases of 801 and 802, and the position of the virtual camera may be set in the middle and the size of the reference object may be adjusted in combination with the adjustment of the zoom value. In this case, it is assumed that it will be easier to set the subsequent virtual camera path to either a position near or far from the reference object.

In FIG. 9, the effect of changing the initial position of the virtual camera according to the position of the generated virtual camera path will be described.

For example, A is defined as a wide-angle camera path, B is defined as an intermediate camera path, C is defined as a telephoto camera path, and the like, the mode is set in the virtual camera path setting unit 509, and the mode is set in the object size calculation unit 505. Set the initial position of the virtual camera according to the mode.

The virtual camera 901 is a virtual camera whose initial position is the position A in FIG. The case of starting the virtual camera path from the position A in FIG. 9 is advantageous when generating the virtual camera path 903 closer to the object close to the reference object.

Further, the virtual camera 902 is a virtual camera whose initial position is the position C in FIG. 9. Starting the virtual camera path from the position C in FIG. 9 is advantageous when generating a virtual camera path 904 with a bird's-eye view far from the reference object.

In the above configuration, the virtual camera path mode sets the distance between the virtual camera and the reference object, but it may also specify the angle of view of the virtual camera. In this case, instead of determining the position of the virtual camera in S606, after determining the angle of view of the virtual camera, the distance between the virtual camera and the reference object is adjusted to make the actual camera image and the reference object the same size. Generate camera parameters.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

10 Virtual viewpoint image generation device 101 Virtual camera operation unit 102 Virtual camera path generation unit 103 Virtual viewpoint image generation unit 104 Virtual viewpoint image image pickup device 105 Object size calculation unit 106 Real camera image image pickup device 107 Real camera image pickup area setting unit 108 Video Switching part

Claims (13)

Parameter acquisition means for acquiring imaging parameters related to an imaging device that images a predetermined object, and
The size acquisition means for acquiring the size of the region corresponding to the predetermined object in the captured image obtained by the imaging performed by the imaging device, the imaging parameters acquired by the parameter acquisition means, and the image acquisition parameters acquired by the size acquisition means. A path generation means for generating a virtual viewpoint path representing a time change of the virtual viewpoint related to the virtual viewpoint image based on the size of the area corresponding to the predetermined object, and
An image generation means for generating a virtual viewpoint image based on a virtual viewpoint path generated by the path generation means and an image obtained by capturing an image of the predetermined object from a plurality of directions.
An output means for selecting and outputting one of the captured image and the virtual viewpoint image generated by the image generation means.
A video processing device characterized by having. The path generation means has a predetermined difference between the size of the region corresponding to the predetermined object in the captured image acquired by the size acquisition means and the size of the region corresponding to the predetermined object in the virtual viewpoint image. Generate the virtual viewpoint path so that it is equal to or less than the threshold value of.
The video processing apparatus according to claim 1. The path generation means has such that the difference in the number of pixels in at least one of the vertical direction and the horizontal direction of the region corresponding to a predetermined object in each image is equal to or less than a predetermined threshold value between the captured image and the virtual viewpoint image. To generate the virtual viewpoint path,
The video processing apparatus according to claim 2. The output means is
When the imaging parameter is within a predetermined range, the captured image is selected and output.
The image processing apparatus according to any one of claims 1 to 3, wherein when the imaging parameter exceeds the predetermined range, the virtual viewpoint image is selected and output. The predetermined range is a range that can be taken by the imaging parameter when it is possible to image the region corresponding to the predetermined object in the captured image so as to have a predetermined size.
The video processing apparatus according to claim 4. When the virtual viewpoint image is output after the captured image by the output means, the path generating means sets the virtual viewpoint at the start position of the virtual viewpoint path based on the image pickup parameter.
The video processing apparatus according to any one of claims 1 to 5. When the captured image is output after the virtual viewpoint image by the output means, the path generating means sets the virtual viewpoint at the end position of the virtual viewpoint path based on the image pickup parameter.
The video processing apparatus according to any one of claims 1 to 6. The size acquisition means specifies the size of a region corresponding to the predetermined object in the captured image based on the distance between the installation position of the image pickup device and the predetermined object based on the image pickup parameter.
The video processing apparatus according to any one of claims 1 to 7. The size acquisition means specifies the size of a region corresponding to the predetermined object in the captured image based on the angle of view of the imaging device based on the imaging parameter.
The video processing apparatus according to any one of claims 1 to 8. It also has a receiving means for accepting user input to specify a virtual viewpoint path.
The path generating means generates the virtual viewpoint path based on the imaging parameter, the size of the region corresponding to the predetermined object, and the user input.
The video processing apparatus according to any one of claims 1 to 9. The imaging parameters include parameters relating to the position, orientation, and angle of view of the imaging device.
The video processing apparatus according to any one of claims 1 to 10. A parameter acquisition step for acquiring imaging parameters related to an imaging device that images a predetermined object, and
A size acquisition step for acquiring the size of a region corresponding to the predetermined object in the captured image obtained by the imaging performed by the imaging device, an imaging parameter acquired by the parameter acquisition step, and an imaging parameter acquired by the size acquisition step. A path generation step for generating a virtual viewpoint path representing a time change of the virtual viewpoint related to the virtual viewpoint image based on the size of the area corresponding to the predetermined object, and
A video generation step for generating a virtual viewpoint image based on a virtual viewpoint path generated by the path generation step and an image obtained by capturing an image of the predetermined object from a plurality of directions.
An output step in which one of the captured image and the virtual viewpoint image generated by the image generation step is selected and output.
A video processing method characterized by having. A program for causing a computer to function as the video processing apparatus according to any one of claims 1 to 11.

Images