JP2024058941A - Control device, control method, and computer program - Google Patents

Abstract

[Problem] When an operator of a virtual camera wishes to manually operate camera parameters that are automatically controlled, there is a risk that the operator will not be able to move the virtual camera as intended. [Solution] Change information for changing the position or attitude of the virtual camera is acquired, and it is determined whether the information is for changing the position of the virtual camera or the attitude of the virtual camera, and the position and attitude of the virtual camera are changed based on the determination result so that a specific object is positioned on the optical axis of the virtual camera. [Selected Figure] Figure 4

Description

The present disclosure relates to a control device, a control method, and a computer program for a virtual camera that generates a virtual viewpoint image.

In recent years, a technology that uses multiple cameras installed in different positions to synchronously capture images from multiple viewpoints and then generates virtual viewpoint images from any viewpoint, rather than just images from the camera installation positions, has been gaining attention. When this technology is applied to sports broadcasts on television and other media, it is possible to generate video content with a high degree of freedom and realism compared to conventionally captured images.

At this time, the operator of the virtual camera must specify the position and orientation of the virtual camera so that the image is appropriate for the scene of the match, specifically the movement of the players and the ball. The virtual camera has three degrees of freedom for both translation and rotation, and can freely manipulate its position and orientation to suit the subject. When shooting a scene in which the virtual camera changes its attitude so that it always faces a specific player or the ball, the operator of the virtual camera must manipulate multiple degrees of freedom to keep the moving subject captured at a desired position within the angle of view. For this reason, there is a demand for simplifying the movement operation, such as manually manipulating only the position of the virtual camera and automatically controlling the attitude of the virtual camera so that it always faces the subject, in order to keep the moving subject captured at a desired position within the angle of view.

Patent document 1 discloses a technology that allows a desired location of a subject to be continuously captured within the field of view of a virtual camera, even in a scene in which the subject moves quickly and in a complex manner, by previously setting camera parameters for automatic control and camera parameters for manual operation.

JP 2021-177351 A

However, because the camera parameters for automatic control and those for manual operation are set in advance, if the camera parameters that the operator (user) wants to manually operate change, there is a risk that the virtual camera will not be able to be moved as the operator intends.

This disclosure was made in consideration of the above-mentioned problems, and aims to enable the operator of the virtual camera to capture the intended virtual viewpoint image with simple operations.

An information processing device according to an embodiment of the present disclosure includes:
an acquisition means for acquiring change information for changing a position or an attitude of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices;
A setting means for setting a particular object;
a determination means for determining whether the change information is information for changing a position of the virtual camera or information for changing an attitude of the virtual camera;
a modification means for modifying a position and an attitude of the virtual camera based on the modification information and a determination result of the determination means so that the specific object is positioned on an optical axis of the virtual camera;
The present invention is characterized by having the following.

According to the present disclosure, the virtual viewpoint image intended by the operator of the virtual camera can be captured with simple operations.

FIG. 1 is a diagram illustrating the configuration of an entire image processing system. FIG. 1 is a configuration diagram of an information processing device. 1 is a diagram showing a world coordinate system and camera parameters and movement operations of a virtual camera. FIG. FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing device. 11 is a diagram showing an example of correspondence between a moving operation of the virtual camera and camera parameters that are automatically controlled during gaze. 5A to 5C are diagrams illustrating an example of correction of camera parameters of a virtual camera according to the first embodiment. 4 is a flowchart showing a process of the image processing device according to the first embodiment.

Embodiments of the present disclosure will be described below with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In each drawing, the same members or elements are given the same reference numbers, and duplicate descriptions are omitted or simplified.

<Embodiment 1>
FIG. 1 is a diagram showing the overall configuration of an image processing system.

The image processing system 10 is composed of an imaging system 101, an image processing device 102, and an information processing device 103. The image processing system 10 is capable of generating a virtual viewpoint image.

The imaging system 101 has multiple physical cameras installed at different positions surrounding the imaging area, and captures images in a time-synchronized manner. Multiple images captured synchronously from multiple viewpoints are transmitted to the image processing device 102. The imaging area may be a photography studio where imaging is performed to generate a virtual viewpoint image, a stadium where a sports competition is held, or a stage where a performance is performed.

The image processing device 102 generates a virtual viewpoint image seen from a virtual camera based on multiple images captured synchronously from multiple viewpoints. The virtual camera is a virtual camera that is different from multiple imaging devices actually installed around the imaging area, and is a concept for conveniently explaining the virtual viewpoint related to the generation of the virtual viewpoint image. That is, the virtual viewpoint image can be considered to be an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the viewpoint in the virtual imaging can be expressed as the position and orientation of the virtual camera. In other words, it can be said that the virtual viewpoint image is an image that simulates an image captured by a camera assuming that a camera exists at the position of the virtual viewpoint set in the space. In addition, in this embodiment, the content of the transition of the virtual viewpoint over time is expressed as a virtual camera path. However, it is not necessary to use the concept of a virtual camera to realize the configuration of this embodiment. That is, it is sufficient that at least information representing a specific position and information representing a direction in the space are set, and the virtual viewpoint image is generated according to the set information. The viewpoint of the virtual camera is expressed by camera parameters determined by the information processing device 103 described later. In the following explanation, unless otherwise specified, the term "image" will be used to include the concepts of both moving images and still images. In other words, the image processing system 10 can process both still images and moving images.

The image processing device 102 extracts the subject as a foreground from the multiple captured images sent from the imaging system 101, and generates a three-dimensional model from the extracted foreground image. One method of extracting the foreground is to use background difference information. In the simplest case, a state in which the foreground does not exist is captured in advance as a background image, and the difference between the image in which the foreground exists and the background image is calculated. If the difference value is greater than a threshold value, the pixel position is determined to be the foreground. There are various other methods for extracting the foreground, such as a method using image features related to the subject or machine learning, but in this proposal, any method for extracting the foreground is acceptable. The three-dimensional model may be generated by generating a 3D point cloud (a set of points having three-dimensional coordinates) using the shape from Silhouette method, or may be generated using depth data obtained from stereo image processing. In this case, the method of generating the three-dimensional model is not limited. A virtual viewpoint image viewed from a virtual camera is generated from this three-dimensional model and a specified background model. The background model may be data of a studio set or a stadium field that has been photographed in advance, or data of a fictitious space generated using CG or the like. Model-Based Rendering (MBR), for example, may be used as a method for generating a virtual viewpoint image. This process makes it possible to generate an image of a three-dimensional model seen from the position and orientation of a virtual camera. Note that the method for generating a virtual viewpoint image is not limited to this.

The information processing device 103 controls the virtual camera and determines camera parameters that represent the viewpoint of the virtual camera. The camera parameters of the virtual camera include parameters for specifying the position, attitude, zoom, or time. The position of the virtual camera specified by the camera parameters is indicated by three-dimensional coordinates with a predetermined position as the origin. The position specified by the camera parameters of the virtual camera is indicated by coordinates of a Cartesian coordinate system of three axes, the X axis, the Y axis, and the Z axis. The attitude of the virtual camera specified by the camera parameters is composed of parameters of three axes, pan, tilt, and roll. The zoom of the virtual camera specified by the camera parameters is indicated by one axis, for example, the focal length. The camera parameters may include parameters that define other elements, and may not include all of the above-mentioned parameters.

The information processing device 103 transmits the determined camera parameters of the virtual camera to the image processing device 102. Next, the image processing device 102 generates a virtual viewpoint image based on the received camera parameters and transmits it to the information processing device 103.

Figure 2(a) is a diagram explaining the hardware configuration of the information processing device 103.

The CPU 201 controls the entire information processing device 103 using computer programs and data stored in the RAM 202 and ROM 203. The information processing device 103 may have one or more dedicated hardware components different from the CPU 201, and at least a portion of the processing by the CPU 201 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

RAM 202 temporarily stores computer programs read from ROM 203, intermediate calculation results, and data supplied from the outside via communication unit 204.

ROM 203 stores computer programs and data that do not require modification.

The communication unit 204 is used for communication between the information processing device 103 and an external device. For example, if the information processing device 103 has a function for wired communication with an external device, the communication unit 204 is provided with a communication means such as Ethernet or USB. If the information processing device 103 has a function for wireless communication with an external device, the communication unit 204 is provided with an antenna.

The input/output unit 205 has multiple input units for controlling the virtual camera and multiple display units for displaying the status of the virtual camera, etc.

The movement control unit 206 controls the movement of the virtual camera and determines the camera parameters based on the contents input to the input/output unit 205 by the operator. In this embodiment, the movement control unit 206 is provided in the information processing device 103, but is not limited to this. It may also be provided in the image processing device 102. In that case, the contents input to the input/output unit 205 are sent to the image processing device 102 as is.

Figure 2 (b) is a diagram explaining the input/output unit 205.

The display unit 211 is composed of, for example, an LCD display or LEDs, and displays a GUI (Graphical User Interface) for the user to operate the information processing device 103, and images including a virtual viewpoint image generated by the image processing device 102.

The input/output unit 205 has joysticks 212a and 212b, seesaw switches 213a and 213b, and a group of buttons 214. The operator operates these as input units to change the camera parameters of the virtual camera. The joysticks 212a and 212b each have camera parameters with three degrees of freedom. In this embodiment, the joystick 212a operates the camera parameters of the X-axis, Y-axis, and Z-axis of the virtual camera, and the joystick 212b operates the camera parameters of the pan, tilt, and roll of the virtual camera. The seesaw switches 213a and 213b change the focal length of the virtual camera and the distance to the point of interest within a predetermined focal length range by tilting them to the plus or minus side. Here, the point of interest is a point on the optical axis that is a certain distance away from the virtual camera. The position of an object that exists on the optical axis of the virtual camera may be set as the point of interest. Alternatively, the intersection of the optical axis of the virtual camera and a plane at a certain height (for example, 1.5 m) from the ground may be set as the point of interest. The gaze area is an area represented by a circle with a radius of a predetermined distance centered on the gaze point. In this embodiment, it is set to 2 m centered on the gaze point. Note that the gaze area is not limited to this, and may be a polygonal area or a spherical area centered on the gaze point. Also, the input/output unit 205 shown in FIG. 2 is merely an example, and the present invention does not limit the configuration of the input/output unit 205.

Figure 3(a) shows the world coordinate system (x, y, z) in the shooting space of the virtual camera.

The world coordinate system is used to represent the camera parameters of the virtual camera and the subject. A subject is a tangible object that exists in the shooting space, such as the field 301, the ball 302, the players 303, etc. In the world coordinate system, the center of the field 301 is set as the origin (0,0,0). The x-axis is set in the direction of the long side of the field 301, the y-axis is set in the direction of the short side of the field 301, and the z-axis is set in the vertical direction relative to the field 301. Note that the method of setting the world coordinate system is not limited to this.

Figure 3(b) shows the camera parameters of the virtual camera.

The camera parameters of the virtual camera are expressed in a Cartesian coordinate system (X, Y, Z) of three axes: X-axis 312, Y-axis 313, and Z-axis 314. The Z-axis 314 is always perpendicular to the field 301, regardless of the attitude of the virtual camera 311. Therefore, the X-axis 312 and Y-axis 313 are always horizontal to the field 301, regardless of the attitude of the camera. The directions of the X-axis 312 and Y-axis 313 change depending on the attitude of the virtual camera 311. The direction in which the optical axis of the virtual camera 311 is projected onto a plane horizontal to the field 301 is the Y-axis 313, and the direction perpendicular to the optical axis of the virtual camera 311 is the X-axis 312. Rotational movements around the X-axis 312, Y-axis 313, and Z-axis 314 as rotation axes are tilt 315, roll 316, and pan 317, respectively. Note that the method of setting the camera parameters is not limited to this.

By combining camera parameters corresponding to the three axes, X-axis 312, Y-axis 313, and Z-axis 314, with camera parameters corresponding to the three axes, tilt 315, roll 316, and pan 317, the virtual camera can freely move and change its posture in the three-dimensional space of the subject (field). Among these, combinations that are easy for the user to operate are prepared as virtual camera movement operations. In this embodiment, operations on camera parameters corresponding to the three axes, X-axis 312, Y-axis 313, and Z-axis 314, are called position operations, and operations on camera parameters corresponding to the three axes, tilt 315, roll 316, and pan 317, are called rotation operations. In addition to the position and rotation operations described above, virtual camera movement operations include zooming in and out, translation with respect to the point of interest, and horizontal and vertical rotation around the point of interest.

Zooming in and out is the operation of enlarging or reducing the size of a subject in the virtual camera's field of view. This is achieved by moving the virtual camera along the optical axis. Moving forward causes the image to zoom in, and moving backward causes the image to zoom out. Note that zooming in and out is not limited to these methods, and it is also possible to change the focal length of the virtual camera, etc.

As shown in FIG. 3(c), translational movement with respect to the point of interest is an operation of moving the position of the point of interest 321 of the virtual camera 311 without changing the attitude of the virtual camera 311. The position of the point of interest is located on the optical axis of the virtual camera. Therefore, when the position of the virtual camera 311 is moved without changing the attitude of the virtual camera 311, the position of the point of interest also moves parallel to the position of the virtual camera 311. Therefore, the movement trajectory of the virtual camera 311 when the position of the virtual camera 311 is changed without changing the attitude of the virtual camera 311 and the movement trajectory of the point of interest 321 are the same movement trajectory.

As shown in FIG. 3(d), horizontal rotation around a point of interest is an operation of rotating the virtual camera 311 around an axis 324 parallel to the Z axis 314, with the point of interest 321 as the center. In other words, it is an operation of moving on a circle with the point of interest 321 as the center. The same can be said for rotation in any direction, so it will not be explained below.

As shown in FIG. 3(e), vertical rotation around a point of interest is an operation of rotating the virtual camera 311 around an axis 326 parallel to the X-axis 312, with the point of interest 321 as the center. In horizontal and vertical rotation around the point of interest, the attitude of the virtual camera changes so that it always faces the point of interest 321, and the coordinates of the point of interest 321 do not change. In addition, the rotation radius is equal to the distance between the virtual camera 311 and the point of interest 321.

By combining horizontal rotation 323 and vertical rotation 325 centered on the attention point 321, it is possible to realize a movement operation of the virtual camera that allows the subject to be viewed from any angle of 360 degrees without changing the coordinates of the attention point 321. Note that the movement operation of the virtual camera is not limited to this, and any operation can be realized by combining the movement of the virtual camera and a change in posture.

Figure 4 is a block diagram showing an example of the functional configuration of an image processing device in embodiment 1. The image processing device 102 receives multiple images captured synchronously from the imaging system 101 and camera parameters from the information processing device 103, and generates a virtual viewpoint image.

The functions of the image processing device 102 are explained in order.

The virtual camera information acquisition unit 401 receives the input contents entered by the operator into the input unit from the input/output unit 205 of the information processing device 103, and acquires the range of the gaze area. It also acquires camera parameters such as the position and attitude of the virtual camera in each frame from the movement control unit 206 of the information processing device 103. These camera parameters are transmitted to the gaze target setting unit 403 and the control determination unit 404.

The subject model generation unit 402 generates model information of the subject based on multiple images received from the imaging system 101 and parameters such as the position and orientation of each imaging device. The model information includes data representing the three-dimensional shape of the model, position information of the model, etc. The position of the model is approximated by the center of gravity of the bounding box that encloses the model, but the method of approximating the model position is not limited to this. The generated model is sent to the virtual viewpoint image generation unit 406.

The gaze target setting unit 403 receives model information from the subject model generation unit 402, and sets the subject corresponding to the input contents received from the input/output unit 205 to the input unit by the operator as the gaze target (specific object), and outputs the model information. For example, when a new keyboard is provided in the input/output unit 205 and the operator presses the right button on the keyboard, the model of the subject located to the right of the currently selected subject model as seen from the virtual camera and having the shortest distance in the X-axis direction is set as the gaze target. The model information of the set subject is transmitted to the gaze target setting unit 403. If there is no input from the input unit to specify the gaze target, the subject closest to the attention point is selected as the gaze target. Note that the gaze target is not limited to a single object, and multiple subjects may be selected as the gaze target. In that case, the position of the model group of the gaze target is approximated by the center of gravity of the bounding box surrounding the model group, but the means for acquiring the position information is not limited to this. The model information selected as the gaze target is transmitted to the control determination unit 404. Note that the gaze target setting unit 403 may be included in a device different from the image processing device 102. For example, the information processing device 103 may have this function, in which case the gaze target is set by the user making an input to select the gaze target.

The control determination unit 404 receives model information of the gaze target from the gaze target setting unit 403 and camera parameters of the virtual camera from the virtual camera information acquisition unit 401, and determines whether automatic control is enabled for the position or posture of the virtual camera so as to gaze at the model of the gaze target. Here, gaze means that the center position of the image captured by the virtual camera coincides with the center of gravity position of the bounding box surrounding the gaze target, or the model of the gaze target is located within the gaze area received from the virtual camera information acquisition unit 401. The determination of whether automatic control is enabled is performed based on the input contents to the input unit by the operator, the relationship between the position of the group of subjects including the gaze target, the position and posture of the virtual camera, and the attention point. In this embodiment, if the gaze target set by the gaze target setting unit 403 cannot be gazed at and the gaze target does not exist within the angle of view of the virtual camera, the automatic control is determined to be enabled. The determination result of whether automatic control is enabled and information indicating the camera parameters to be subject to automatic control are transmitted to the position and posture control unit 405. If automatic control is determined to be enabled, a determination is made as to whether or not to cancel automatic control from the next frame. The determination as to whether or not to cancel automatic control is made based on the input by the operator to the input unit. Specifically, automatic control is canceled if there is an input for a camera parameter that is the subject of automatic control. In other words, if the pilot wishes to cancel automatic control, he or she can perform an operation to change the automatically controlled parameter. The result of the determination as to whether or not to cancel automatic control is sent to the position and orientation control unit 405. Note that, because automatic control is a process of modifying camera parameters, the control determination may also be called a modification determination or a change determination.

When the control determination unit 404 determines that automatic control is enabled, the position and orientation control unit 405 automatically controls some of the camera parameters to focus on the model selected as the gaze target, and modifies the camera parameters of the virtual camera. At this time, the camera parameters to be automatically controlled are determined according to the movement operation input to the input unit by the operator and the movement of the gaze target model. In other words, the camera parameters to be automatically controlled are determined by the input of the operator when automatic control is enabled. The modified camera parameters are sent to the virtual viewpoint image generation unit 406. Automatic control that is determined to be enabled continues until it is determined to be released. When automatic control is enabled and information to release the automatic control is received from the control determination unit 404, the automatic control is released.

The virtual viewpoint image generation unit 406 acquires the subject model information from the subject model generation unit 402 and the camera parameters of the virtual camera from the position and orientation control unit 405, renders the subject model as viewed from the position and orientation of the virtual camera, and generates a virtual viewpoint image. The generated virtual viewpoint image is transmitted to the information processing device 103 and displayed on the display unit of the input/output unit 205.

Figure 5 (a) is a diagram showing an example of the correspondence between the operation contents of the virtual camera and the camera parameters that are automatically controlled when gazing. When the position and attitude of the virtual camera are automatically controlled to gaze at the model of the gaze target, the camera parameters that are automatically controlled to gaze are determined according to the operation contents of the virtual camera by the operator inputted to the input unit. However, the camera parameters that are automatically controlled are different from the camera parameters of the movement operation inputted to the input unit. In addition, the movement operation of the virtual camera by the operator and the camera parameters that are automatically controlled when gazing are not necessarily in one-to-one correspondence, and there may be two or more candidates for the camera parameters that are automatically controlled for one movement operation of the virtual camera. In this case, when the virtual camera is moved to gaze at, the camera parameters of the movement that have a small change amount of the camera parameters are automatically controlled. Rotation around the X axis Furthermore, when multiple movement operations are inputted to the input unit, the camera parameters that are automatically controlled are determined so that the total number of camera parameters that are automatically controlled is minimized. Note that the method of determining the camera parameters to be automatically controlled from the candidates of the camera parameters is not limited to this.

In this embodiment, if the operation is translation in the X-axis direction, the pan is automatically controlled. If the operation is translation, the operator performs an operation to specify the position of the virtual camera, and thus the position of the virtual camera can be grasped. However, if the operation is rotation, the camera parameters corresponding to the position of the virtual camera are automatically controlled, and there is a risk that the operator will not be able to grasp where the position of the virtual camera will actually move. Therefore, in this embodiment, if the operation is rotation, automatic control is performed to move the position of the virtual camera on a circle with the gaze target as the center and the distance between the gaze target and the virtual camera as the radius. In this way, even if the operation is rotation, it is easy to imagine where the position of the virtual camera will move. Note that this is not limited to this, and if the operation is rotation, the position of the virtual camera may be automatically controlled to move in a translational manner. In addition, a separate button may be provided on the controller, and the content of the automatic control may be switched by pressing the button. For example, if the content of the automatic control is rotation around the gaze target, pressing the button can switch to translation.

Figure 5(b) shows the classification of translational and rotational movements of the virtual camera. In this embodiment, translational movement of the virtual camera refers to movement in which only the position of the virtual camera is changed, without changing the attitude. In other words, it is to change the parameters of the X-axis, Y-axis, and Z-axis, and not to change the parameters of the pan, tilt, and roll. Since the attitude is not changed, the position of the virtual camera and the position of the point of interest move in parallel, so it is defined as translational movement. Note that rotational movement of the virtual camera refers to an operation in which only the attitude of the virtual camera is changed, without changing the position. In other words, it is to change the parameters of the pan, tilt, and roll, and not to change the parameters of the X-axis, Y-axis, and Z-axis. The correspondence between the movement operation of the virtual camera by the operator and the camera parameters that are automatically controlled when gazing is not limited to the correspondence shown in Figure 5(a), and it is sufficient that one is translational movement and the other is rotational movement. In other words, the movement operation by the operator and the camera parameters that are automatically controlled are not both translational movement (or rotational movement). Note that enlargement and reduction are considered as translational movement in the optical axis direction, and are included in translational movement. Horizontal rotation around a point of interest is included in rotational movement.

On the other hand, if the model being watched moves even when the virtual camera is stopped, the relative positional relationship between the virtual camera and the object being watched changes. For this reason, camera parameters that are subject to automatic control when the model being watched moves while the virtual camera is stopped are set in advance. Specifically, camera parameters corresponding to either the position or the attitude of the virtual camera are set. In this embodiment, camera parameters corresponding to the position of the virtual camera are set. The operator can set which control is to be performed, and the controller is provided with a button (not shown) that switches the object of automatic control between translational movement and rotational movement when the virtual camera is stopped.

Figure 6 is a diagram showing an example of how camera parameters of a virtual camera are modified in embodiment 1. When the control determination unit 404 determines that automatic control should be performed, the position and orientation control unit 405 modifies the camera parameters based on the input content by the operator via the input unit, the position of the group of subjects including the gaze target, the position and orientation of the virtual camera, and the relationship between the attention points. When automatic control is active, some of the camera parameters are automatically controlled in response to the movement operation input by the operator via the input unit and the movement of the model of the gaze target. Note that when there is an input for the automatically controlled camera parameters, the automatic control is released.

6A is a diagram showing an example of camera parameter correction based on the positional relationship between the model 602 of the gaze target and the attention point 603 of the virtual camera 601. After the operator performs a horizontal rotation movement operation 604 around the attention point 603 and a movement operation combining these, the position of the model 602 of the gaze target may move away from the attention point 603 due to the movement of the gaze target. In this case, when the model 602 of the gaze target moves from inside to outside the gaze area 604 in the image captured by the virtual camera 601, the control determination unit 404 determines that automatic control is to be performed on the virtual camera. When automatic control is enabled, the model information of the gaze target and the camera parameters of the virtual camera in the frame in which automatic control is enabled are referenced to determine the parameters for automatic control. In FIG. 6A, it is assumed that the gaze target 602 moves outside the gaze area 604 when the operator finishes the horizontal rotation around the attention point 603 and the operator is not operating the virtual camera. At this time, the virtual camera translates 605 so that the position coordinates of the attention point 603 match those of the gazed-upon model 602. Note that the method of matching the position coordinates of the attention point 603 and the gazed-upon model 602 is not limited to this. While automatic control is active, the position coordinates of the attention point 603 are updated for each frame and the virtual camera translates 605 so that the position coordinates of the attention point 603 and the gazed-upon model 602 always match. At this time, the value of the distance between the attention point and the virtual camera immediately after automatic control is enabled is stored, and while automatic control is active, the virtual camera translates 605 so that the distance between the attention point 603 and the virtual camera 601 always becomes the stored value. However, if the operator performs a zoom in or zoom out operation, the virtual camera is translated 605 so that the changed value is obtained.

6B is a diagram showing an example of camera parameter correction based on the relationship between the position of the group of objects including the gaze target and the position and attitude of the virtual camera. When the gaze target model 602 moves from inside to outside the gaze area 604 in the captured image of the virtual camera 601 due to the operator's movement operation 606 of the virtual camera and the change in the position of the gaze target model 602, automatic control is enabled and the camera parameters are corrected. When automatic control is enabled, the virtual camera rotates 608 to gaze at the gaze target model 602. Here, when automatic control is enabled because the gaze target model 602 is blocked by a subject 607 that is not the gaze target, the virtual camera rotates 608 by horizontal rotation around the attention point 603 to gaze at the gaze target. While automatic control is enabled, the position information of the gaze target model received from the gaze target setting unit 403 and the camera parameters of the virtual camera received from the virtual camera information acquisition unit 401 are referenced to determine whether the gaze target model is being gazed at for each frame. If it is determined that the object is not being gazed at, the virtual camera is translated or rotated 608 so as to gaze at the model of the object of gaze. In this embodiment, the virtual camera is rotated 608 to a position where the object of gaze is not blocked. Furthermore, the virtual camera may be rotated 608 until the object 607 that is not the object of gaze is a predetermined distance away from the optical axis of the virtual camera.

Figure 6 (c) is a diagram showing an example of camera parameter correction based on the movement operation of the virtual camera by the operator. When the control determination unit 404 determines that automatic control is to be performed, some camera parameters are automatically controlled according to the movement operation input to the input unit and the movement of the model of the gaze target. When an input is made to the automatically controlled camera parameters while the automatic control is enabled, the automatic control is released. For example, when translational movement 609 in the X-axis and Y-axis directions is input, pan corresponds to the translational movement in the X-axis direction, and tilt or pan corresponds to the translational movement in the Y-axis direction. When multiple movement operations are input to the input unit, the automatically controlled camera parameters are determined so that the total number of automatically controlled camera parameters is minimized, so that the automatically controlled camera parameters are panned. Next, the movement operation is changed from translational movement 609 in the X-axis and Y-axis directions to horizontal rotation 610 around the point of interest. At this time, since the operation when the automatic control was enabled was translational movement 609 in the X-axis and Y-axis directions, panning is automatically controlled. Because the operator has input horizontal rotation 610 around the point of interest, the input parameters are X and Y, which specify the position of the virtual camera, as well as pan, which specifies the attitude of the virtual camera. At this time, because the input includes pan, it is determined that an input has been made to a parameter that was automatically controlled, and the automatic control is released.

Figure 7 is a flowchart showing the processing of the image processing device in embodiment 1.

In step S701, the subject model generation unit 402 acquires multiple images captured from different directions by the imaging system 101 and camera parameters of the position and orientation of the imaging device, and generates a subject model.

In step S702, the virtual camera information acquisition unit 401 acquires camera parameter information such as the position and attitude of the virtual camera from the virtual camera movement operation input via the input unit of the information processing device 103. This camera parameter information is obtained by converting the operator's input information into camera parameters, and can therefore also be considered change information or update information for changing the initial values or the camera parameters of the previous frame. The acquired camera parameter information is transmitted to the control determination unit 404.

In step S703, the control determination unit 404 determines whether or not automatic control is enabled. If automatic control is enabled, the process proceeds to step S704. If automatic control is not enabled, the process proceeds to step S706.

In step S704, the control determination unit 404 refers to the camera parameter information acquired in step S702 and determines whether it includes input for the camera parameters under automatic control. If it includes input for the camera parameters under automatic control, the process proceeds to step S705. If it does not include input for the camera parameters under automatic control, the process proceeds to step S706.

In step S705, the control determination unit 404 cancels the enabled automatic control. At this time, information to cancel the automatic control is sent to the position and orientation control unit. For example, a parameter auto is provided to determine whether or not automatic control is enabled, and information is sent indicating that automatic control is enabled when auto=1 and disabled when auto=0.

In step S706, the gaze target setting unit 403 acquires an input from the operator to select a gaze target from the input unit of the information processing device 103. In this embodiment, a subject ID corresponding to the subject model generated in S701 is generated in advance, and the operator sets the gaze target by inputting the subject ID of the gaze target. If there is no input to select a gaze target, the subject closest to the attention point is set as the gaze target.

In step S707, the gaze target setting unit 403 refers to the position information of the subject model generated by the subject model generation unit 402, and obtains the model information of the subject that matches the position information of the subject selected in step S706.

In step S708, the control determination unit 404 refers to the camera parameters of the virtual camera acquired in step S702 and the model information of all subjects including the gaze target generated by the subject model generation unit 402, and determines whether the automatic control activation condition is met. If the activation condition is met, the process proceeds to step S709, and if not, the process proceeds to step S714. Note that if the parameter auto indicating whether automatic control is enabled is auto=1, the process proceeds to step S709 without determining the automatic control activation condition.

In step S709, the control determination unit 404 determines whether the gaze target is obstructed in the captured image by a subject that is not the gaze target and is located between the virtual camera and the gaze target model. If the gaze target is obstructed, the process proceeds to step S713; if not, the process proceeds to S710.

In step S710, the virtual camera information acquisition unit 401 determines whether the virtual camera movement operation acquired in step S702 is a translational movement. If it is a translational movement, the process proceeds to step S712; if not, the process proceeds to step S711.

In step S711, the position and orientation control unit 405 translates the virtual camera so as to gaze at the model of the gaze target, and corrects the camera parameters, based on the camera parameters acquired from the virtual camera information acquisition unit 401. Furthermore, if the virtual camera information acquired in step S702 is the same as that of the previous frame, the camera parameters are automatically controlled in accordance with the movement of the model of the gaze target. Specifically, when the gaze target moves, the virtual camera is translated so as to gaze at the model of the gaze target, and the camera parameters are corrected. The corrected camera parameters are then transmitted to the virtual viewpoint image generation unit 406.

In step S712, the position and orientation control unit 405 rotates and moves the virtual camera so that the virtual camera gazes at the model of the gaze target, thereby correcting the camera parameters acquired from the virtual camera information acquisition unit 401. The position and orientation control unit 405 then transmits the corrected camera parameters to the virtual viewpoint image generation unit 406.

In step S713, the position and orientation control unit 405 corrects the camera parameters acquired from the virtual camera information acquisition unit 401 by moving the virtual camera by horizontal and vertical rotation around the attention point so as to gaze at the model of the gaze target. The corrected camera parameters are then transmitted to the virtual viewpoint image generation unit 406.

In step S714, the virtual viewpoint image generation unit 406 acquires the model information of the subject generated in step S701 and the camera parameters acquired in step S702 or the camera parameters modified in steps S711 to S713. Then, rendering of the subject model as viewed from the position and orientation of the virtual camera is performed to generate a virtual viewpoint image. At this time, if no input for selecting the gaze target has been acquired in step S706, the camera parameters acquired in step S702 are used, and if input has been acquired, the camera parameters modified in steps S711 to S713 are used.

In step S715, it is determined whether an instruction to end the work on the information processing device has been given. If an instruction to end the work has not been given, the process returns to step S701 and the flow from step S701 to step S715 is repeated, and if an instruction to end the work has been given, the flow ends.

By the above process, parameters not operated by the operator are automatically controlled according to the operation contents of the operator when the automatic control is enabled, and the automatic control continues until the automatic control is released. This process can release the automatic control even if the camera parameters that the operator wants to operate change, and the virtual viewpoint image that the operator intends can be generated.

In this embodiment, the processing is performed for each frame, but the operator may specify the time during which the automatic control of gazing at the gaze target is effective, and the processing may be performed only during that time.

In this embodiment, model information for the selected gaze target is read, and the position and attitude of the virtual camera are controlled to gaze at the gaze target based on the input contents by the operator to the input unit, the positions of the group of subjects including the gaze target, the position and attitude of the virtual camera, and the relationship of the point of interest. At this time, a process of gaze is performed by automatically controlling preset camera parameters according to the input contents, so that the operator can easily predict the camera parameters to be automatically controlled, and it is possible to prevent operational errors when automatic control by automatic control and manual operation are performed at the same time. In addition, automatic control of the camera parameters of the virtual camera is performed only when the subject to be gazed at is not gazed at, so there is no excessive reduction in the degree of freedom of camera movement, which is an advantage of virtual viewpoint image capture.

<Embodiment 2>
In the first embodiment, a process was described in which the position and attitude of the virtual camera are controlled to gaze at the target of gaze based on the contents of the input to the input unit by the operator, the position of the group of subjects including the target of gaze, the position and attitude of the virtual camera, and the relationship of the point of interest. At this time, in the camera parameters that are automatically controlled for gaze, if there is an input to the input unit by the operator for the automatically controlled camera parameters, the automatic control is released. However, there is a possibility that the operator wants to adjust the camera parameters during automatic control without releasing the automatic control. For example, in the first embodiment, the camera is automatically controlled to gaze at the target of gaze during automatic control, but there is a possibility that the operator wants to generate a virtual viewpoint image in which the target of gaze is at the edge of the gaze area. Therefore, in the second embodiment, it is assumed that the camera parameters that are the target of automatic control can be changed without releasing the automatic control if the input is equal to or less than a predetermined value.

If the input amount to the input unit for the automatically controlled camera parameters is small, the automatic control is not disabled and is reflected in the automatically controlled parameters. For example, if the operator inputs a translational movement and the rotational movement is automatically controlled, the operator inputs a rotational movement (pan) around the X axis. At this time, if the input amount is less than a predetermined value, the automatic control is not released and the parameters are updated. The predetermined value is set in advance, and in this embodiment, if the input amount is less than 0.5 m/s, the automatic control is not released. If the automatic control is not released, the input operation is reflected in the parameters to be automatically controlled until the automatic control is released. In other words, the position of the gaze target reflected in the virtual viewpoint image during automatic control can be adjusted, and the gaze target can be adjusted to the operator's desired position. The control determination unit 404 determines whether the input amount is small or not, and if the input amount input to the input unit is less than a reference value set in advance by the operator, it is determined to be small. The operator inputs a reference value through the input unit, and the control determination unit 404 stores and refers to the numerical value. The method of determining whether the input amount is small or not is not limited to this. For example, whether the input amount is small may be determined by relatively comparing the input amount to the input unit for the automatically controlled camera parameter with the input amount for the other camera parameters. It is possible to control the camera parameters by applying a weighting coefficient to the input to the input unit. In this case, the camera parameters may be changed without canceling the automatic control if the input is such that the change amount of the camera parameter is equal to or less than a predetermined value. This makes it easier for the operator to input the virtual camera.

A computer program that realizes all or part of the control in this embodiment and the functions of the above-described embodiment may be supplied to an image processing system or the like via a network or various storage media. A computer (or a CPU, MPU, etc.) in the image processing system or the like may then read and execute the program. In this case, the program and the storage medium on which the program is stored constitute the present disclosure.

The disclosure of this embodiment includes the following configurations, methods, and programs.

(Configuration 1)
an acquisition means for acquiring change information for changing a position or an attitude of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices;
a determination means for determining whether the change information is information for changing a position of the virtual camera or information for changing an attitude of the virtual camera;
a modification means for, when the modification information is information for modifying a position of the virtual camera, modifying an attitude of the virtual camera so that a specific object is located on an optical axis of the virtual camera while modifying a position of the virtual camera based on the modification information, and for, when the modification information is information for modifying an attitude of the virtual camera, modifying a position of the virtual camera so that the specific object is located on the optical axis of the virtual camera while modifying the attitude of the virtual camera based on the modification information;
An apparatus comprising:

(Configuration 2)
The device described in configuration 1, characterized in that, when the change information is information for changing the attitude of the virtual camera, the change means changes the attitude of the virtual camera based on the change information, and further changes the position of the virtual camera so as to keep a constant distance between the specific object and the virtual camera.

(Configuration 3)
3. The apparatus according to claim 1, further comprising a change determining means for determining whether or not the change is to be made by the change means.

(Configuration 4)
The device described in configuration 3, wherein the change determination means determines that a change will be made by the change means when the specific object is not located at a predetermined distance from the virtual camera's point of view, and determines that a change will not be made by the change means when the specific object is located at a predetermined distance from the virtual camera's point of view.

(Configuration 5)
The obtaining means obtains a three-dimensional model of an object;
The device according to any one of configurations 1 to 4, wherein the modification means modifies the position and orientation of the virtual camera so that the center of gravity of the three-dimensional model of the specific object is located on the optical axis of the virtual camera.

(Configuration 6)
The position of the virtual camera is expressed by X-axis, Y-axis, and Z-axis parameters,
The apparatus according to any one of configurations 1 to 5, wherein the attitude of the virtual camera is expressed by pan, tilt, and roll parameters.

(Configuration 7)
2. The information processing apparatus according to claim 1, further comprising setting means for setting a specific object.

(Configuration 8)
The apparatus according to configuration 7, wherein the setting means sets the specific object based on an operation by an operator.

(Configuration 9)
9. The apparatus according to configuration 8, wherein the setting means sets the specific object based on an input specifying the specific object from among a plurality of objects.

(Configuration 10)
The apparatus according to configuration 7, wherein the setting means sets an object that is closest to a point of interest of the virtual camera as the specific object.

(Configuration 11)
the determining means determines whether or not an object other than the specific object is located on an optical axis connecting the specific object and the virtual camera;
The device described in any one of configurations 1 to 10, characterized in that the modification means changes the position and orientation of the virtual camera so that an object other than the specific object is not located on an optical axis connecting the specific object and the virtual camera.

(Configuration 12)
The device described in configuration 11, characterized in that the modification means modifies the position and orientation of the virtual camera so as to move on a circle having the position of the specific object as its center and the distance from the position of the specific object to the virtual camera as its radius.

(Configuration 13)
13. The device according to any one of configurations 1 to 12, wherein the change information changes the position or attitude of the virtual camera based on a joystick input.

(Configuration 14)
14. The apparatus according to any one of claims 1 to 13, further comprising a display means for displaying a virtual viewpoint image generated based on the position and attitude of the virtual camera changed by the change means.

(Configuration 15)
the change information includes a change amount of the position of the virtual camera and a change amount of the attitude of the virtual camera,
15. The device according to any one of configurations 1 to 14, wherein the change means changes the position or attitude of the virtual camera based on the change information when the amount of change is equal to or less than a predetermined value.

(Configuration 16)
an acquisition means for acquiring first change information for changing a position or an attitude of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices;
A setting means for setting a particular object;
a determination means for determining whether the first change information is information for changing a position of a virtual camera or information for changing an attitude of the virtual camera;
an output means for outputting second modification information for modifying a position and an attitude of the virtual camera based on the first modification information and a determination result of the determination means so that the specific object is positioned on an optical axis of the virtual camera;
An apparatus comprising:

(Method)
an acquisition step of acquiring change information for changing a position or an attitude of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices;
a configuration step for configuring a particular object;
a determination step of determining whether the change information is information for changing a position of a virtual camera or information for changing an attitude of the virtual camera;
a modifying step of modifying a position and an attitude of the virtual camera based on the modification information and a determination result of the determining means so that the specific object is located on an optical axis of the virtual camera;
13. An information processing method comprising:

(program)
16. A program for controlling each means of the device according to any one of configurations 1 to 15 by a computer.

401 Virtual camera information acquisition unit 403 Gaze target setting unit 404 Control determination unit 405 Position and orientation control unit

Claims (18)

an acquisition means for acquiring change information for changing a position or an attitude of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices;
a determination means for determining whether the change information is information for changing a position of a virtual camera or information for changing an attitude of the virtual camera;
a modification means for, when the modification information is information for modifying a position of the virtual camera, modifying an attitude of the virtual camera so that the specific object is located on an optical axis of the virtual camera while modifying a position of the virtual camera based on the modification information, and for, when the modification information is information for modifying an attitude of the virtual camera, modifying a position of the virtual camera so that the specific object is located on the optical axis of the virtual camera while modifying the attitude of the virtual camera based on the modification information;
13. An information processing device comprising: The information processing device according to claim 1, characterized in that, when the change information is information for changing the attitude of the virtual camera, the change means changes the attitude of the virtual camera based on the change information, and further changes the position of the virtual camera so as to keep a constant distance between the specific object and the virtual camera. The information processing device according to claim 1, further comprising a change determination means for determining whether or not to make a change by the change means. The information processing device according to claim 3, characterized in that the change determination means determines that the change is to be made by the change means when the specific object is not located at a predetermined distance from the virtual camera's point of view, and determines that the change is not to be made by the change means when the specific object is located at a predetermined distance from the virtual camera's point of view. The obtaining means obtains a three-dimensional model of an object;
2 . The information processing apparatus according to claim 1 , wherein the change means changes the position and attitude of the virtual camera so that the center of gravity of the three-dimensional model of the specific object is positioned on the optical axis of the virtual camera. The position of the virtual camera is expressed by X-axis, Y-axis, and Z-axis parameters,
2 . The information processing apparatus according to claim 1 , wherein the attitude of the virtual camera is expressed by pan, tilt, and roll parameters. The information processing device according to claim 1 further comprises a setting means for setting a specific object. The information processing device according to claim 7, characterized in that the setting means sets the specific object based on an operation by an operator. The information processing device according to claim 8, characterized in that the setting means sets the specific object based on an input that specifies the specific object from among a plurality of objects. The information processing device according to claim 7, characterized in that the setting means sets the object closest to the focus point of the virtual camera as the specific object. the determining means determines whether or not an object other than the specific object is located on an optical axis connecting the specific object and the virtual camera;
2 . The information processing apparatus according to claim 1 , wherein the change means changes the position and attitude of the virtual camera so that an object other than the specific object is not positioned on an optical axis connecting the specific object and the virtual camera. The information processing device according to claim 11, characterized in that the change means changes the position and orientation of the virtual camera so that it moves on a circle whose center is the position of the specific object and whose radius is the distance from the position of the specific object to the virtual camera. The information processing device according to claim 1, characterized in that the change information changes the position or attitude of the virtual camera based on a joystick input. The information processing device according to claim 1, further comprising a display means for displaying a virtual viewpoint image generated based on the position and orientation of the virtual camera changed by the change means. the change information includes a change amount of the position of the virtual camera and a change amount of the attitude of the virtual camera,
2 . The information processing apparatus according to claim 1 , wherein the change means changes the position or the attitude of the virtual camera based on the change information when the amount of change is equal to or smaller than a predetermined value. an acquisition means for acquiring first change information for changing a position or an attitude of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices;
A setting means for setting a particular object;
a determination means for determining whether the first change information is information for changing a position of a virtual camera or information for changing an attitude of the virtual camera;
an output means for outputting second modification information for modifying a position and an attitude of the virtual camera based on the first modification information and a determination result of the determination means so that the specific object is positioned on an optical axis of the virtual camera;
13. An information processing device comprising: an acquisition step of acquiring change information for changing a position or an attitude of a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices;
a configuration step for configuring a particular object;
a determination step of determining whether the change information is information for changing a position of a virtual camera or information for changing an attitude of the virtual camera;
a modifying step of modifying a position and an attitude of the virtual camera based on the modification information and a determination result of the determining means so that the specific object is located on an optical axis of the virtual camera;
13. An information processing method comprising: A computer program for controlling each means of the image processing device according to any one of claims 1 to 15 by a computer.

Images