JP2023019727A - Imaging control system and operation method of the same - Google Patents

Abstract

To provide an imaging control system which tracks a subject continuously, and an operation method of the system.SOLUTION: The imaging control system comprises: shape estimation means 202 which estimates a three-dimensional shape of a subject from a plurality of captured images captured by a plurality of imaging apparatuses, and generates model information of the subject; a virtual viewpoint image generation unit 209 which generates a virtual viewpoint image based on the captured images and the model information; and a camera mechanism control unit 206 which performs control to vary a positional attitude of at least one imaging means different from the imaging apparatuses based on the model information.SELECTED DRAWING: Figure 2

Description

The present invention relates to an imaging control system and its operating method.

In recent years, multiple cameras have been installed at different positions to take synchronous multi-view images. The technology to generate it is attracting attention. According to the above technology, for example, the highlight scenes of soccer or basketball can be viewed from various positions and angles, such as viewing from the field during a game, which is not possible with the conventional technology. By comparison, the user can obtain a high sense of realism.

In generating and browsing virtual viewpoint images based on multi-viewpoint images, images captured by a plurality of cameras are aggregated in an image processing device such as a server. Then, the image processing apparatus performs processes such as foreground/background separation, 3D shape estimation, 3D shape data generation, and rendering, and transfers the video to the user terminal.

On the other hand, sports broadcasts and the like are shot using various mobile cameras in order to capture realistic images. For example, there are crane cameras and cameras called spider cams (registered trademark), and these cameras require high skill to operate.

On the other hand, Japanese Patent Laid-Open No. 2002-200002 discloses a technique for automatically tracking a target subject and assisting in photographing with respect to these cameras.

JP-A-2005-86360

However, with the technique described in Patent Document 1, it is difficult to continue tracking when the subject disappears from within the angle of view of the camera or when the subject exceeds the range that can be tracked by the camera.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for continuously tracking a subject.

A photographing control system according to the present invention for achieving the above object includes:
a shape estimating means for estimating a three-dimensional shape of a subject from a plurality of captured images captured by a plurality of imaging devices and generating model information of the subject;
virtual viewpoint image generation means for generating a virtual viewpoint image based on the plurality of captured images and the model information;
control means for controlling to change the position and orientation of at least one photographing means different from the plurality of photographing devices based on the model information;
characterized by comprising

According to the present invention, the subject can be continuously tracked.

The figure which shows an example of the hardware constitutions of the imaging|photography control system which concerns on one Embodiment. 4 is a diagram showing an example of functional blocks of the imaging control system according to the first embodiment; FIG. 4 is a flow chart showing a procedure of processing when a mobile camera tracks a subject in the shooting control system according to the first embodiment; 4A and 4B are schematic diagrams showing how a subject is specified according to the first embodiment; FIG. 4 is a schematic diagram showing how the wire camera according to the first embodiment tracks a subject; FIG. FIG. 4 is a diagram showing how camera parameters are obtained when a collision is predicted according to the first embodiment; The figure which shows an example of the functional block of the imaging|photography control system which concerns on Embodiment 2. FIG. 10 is a flowchart showing the procedure of processing when the mobile camera tracks a subject in the imaging control system according to the second embodiment; FIG. 8 is a schematic diagram showing how a crane camera according to the second embodiment tracks a ball; The figure which shows an example of the functional block of the imaging|photography control system which concerns on Embodiment 3. FIG. 10 is a flowchart showing the procedure of processing when tracking is performed while switching cameras in the imaging control system according to the third embodiment; FIG. 11 is a schematic diagram showing a soccer match according to Embodiment 3; 10 is a flowchart showing a procedure of switching processing based on a user's operation or a signal from an external device in the imaging control system according to the fourth embodiment; FIG. 12 is a schematic diagram showing a soccer match according to Embodiment 4; 14 is a flowchart showing a procedure of switching processing during avoidance processing of the mobile camera in the shooting control system according to the fifth embodiment; FIG. 11 is a schematic diagram showing a part of a scene of a soccer match according to Embodiment 5;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(Embodiment 1)
In this embodiment, the model information of the subject used to generate the virtual viewpoint image is used to control the imaging device that performs imaging while physically moving in the real space. will be explained.

<Hardware configuration>
FIG. 1 is a diagram showing an example of a hardware configuration of an imaging control system 100 according to one embodiment.

The imaging control system 100 includes a CPU (Central Processing Unit) 101 , a main storage device 102 , an auxiliary storage device 103 , an external I/F 104 and a GPU (Graphics Processing Unit) 105 . They are connected via an internal bus 108 so as to be able to communicate with each other.

A CPU 101 is one or more central processing units, and controls the overall imaging control system 100 such as various calculations and judgments, image processing, and data input/output.

The main storage device 102 is a storage device that functions as a work area for the CPU 101 and a temporary storage area for data. The main storage device 102 is implemented using storage media such as Dynamic Random Access Memory (DRAM) and Static Random Access Memory (SRAM).

The auxiliary storage device 103 is a storage device that stores information such as various programs, various setting information, various image data, camera parameters, three-dimensional shape data, and two-dimensional maps. The auxiliary storage device 103 is implemented using storage media such as various types of read only memory (ROM), nonvolatile memory such as flash memory, hard disk drive (HDD), solid state drive (SSD), and tape media. A plurality of them may be combined in order to increase capacity and speed, or may be logically integrated into one such as RAID (Redundant Array of Inexpensive Disks).

The external I/F 104 is a communication interface used for communication with external devices such as cameras, PCs, and servers. Alternatively, it is a connection interface with an external device such as a joystick, keyboard, or mouse that accepts various operations by the user. Alternatively, it is a general-purpose input/output interface for connecting an external storage and performing file input/output. The external I/F 104 is implemented by an interface having a physical cable connection terminal such as InfiniBand, Ethernet, or USB, or a wireless interface such as wireless LAN or Bluetooth (registered trademark). USB is an abbreviation for Universal Serial Bus.

The GPU 105 is an arithmetic unit for executing image processing operations at high speed, and has a function of outputting a video signal to an external video display device such as a television. In addition to the process of rendering an image from a virtual viewpoint, it also has a function of generating shape data and color information necessary for a virtual viewpoint image. A video signal output from the GPU 105 is implemented as a video terminal such as DVI, HDMI (registered trademark), Display Port, or SDI. Note that the GPU 105 may include a separate main memory device different from the main memory device 102 . Also, the GPU 105 may perform part or all of the various operations performed by the CPU 101 other than those described above. The GPU 105 does not have a video signal output function, and the external I/F 104 further has a video output function or a data output function necessary for video output. A configuration in which only computation of color information or the like may be performed.

The photographing device 106 has at least two or more cameras each including an optical system such as a lens and an optical sensor such as a CMOS sensor, and converts a photographed image into a signal.

The mechanical device 107 is a device that can mechanically change its shape by a driving force such as a motor. A part or all of the photographing device 106 is installed in the mechanical device 107, and the photographing position and posture of the camera can be mechanically changed.

The internal bus 108 is configured by a general-purpose bus such as PCI Express, and can perform bidirectional communication with each part of the hardware block.

In the imaging control system 100, the CPU 101 reads out one or more programs stored in the auxiliary storage device 103 and executes processing, thereby realizing image processing in flowcharts to be described later.

Note that the configuration shown in FIG. 1 is an example of configuring the imaging control system 100 . For example, the functions shown in FIG. 1 may be realized by separate hardware, and for example, the auxiliary storage device 103 may be connected to the outside of the imaging control system 100 via the external I/F 104 . Also, the main storage device 102 and the auxiliary storage device 103 may be realized by one piece of hardware. Further, instead of the GPU 105, other hardware such as FPGA, DSP, image processing LIS, etc. may be used without departing from the gist of the embodiments. FPGA is an abbreviation for Field-Programmable Gate Array. DSP is an abbreviation for Digital Signal Processor. The internal bus 108 may use any bus as long as each part of the hardware block can communicate bidirectionally. A non-standardized proprietary design communication bus may also be used. Also, the imaging control system 100 is not limited to one device. A plurality of devices having the same configuration may be prepared, and each device may have a different configuration without departing from the spirit of the embodiments.

<Functional configuration>
FIG. 2 is an example of functional blocks of the imaging control system 100 according to this embodiment.

The virtual viewpoint image generation camera unit 201 is composed of at least two or more cameras installed at physically separated locations (for example, installed so as to surround a stadium). Shoot from the other side. The cameras used here are synchronized, the shooting timing of the same time code is controlled with high accuracy, and simultaneous shooting is possible.

A shape estimation processing unit 202 separates an image captured by the virtual viewpoint image generation camera unit 201 into a foreground and a background, and performs shape estimation processing necessary for generating a virtual viewpoint image in real time. The shape estimation process is a process of generating and acquiring model information for generating a virtual viewpoint image using camera parameters including information such as the position and orientation of each camera. It can include dimensional shape data, their color information, world coordinate information, and the like.

The subject extraction unit 203 extracts designated three-dimensional shape data from the shape estimated by the shape estimation processing unit 202 . For example, when data of a virtual viewpoint image to be generated targets sports such as soccer, an object having a characteristic size and shape such as a ball is extracted as a specific subject. Alternatively, a specific person model is extracted from the bounding box of individual model data, the clothing including face information, uniform number, and the like. Alternatively, a crane camera or the like is extracted by attaching a marker for identifying the subject. That is, the subject extraction unit 203 performs processing for extracting the estimated three-dimensional shape as individual subjects.

A user operation unit 204 is configured by the external I/F 104 and the like, and receives input from the user through a user interface.

The mobile camera unit 205 is a camera that is installed with a mechanical structure such as a crane or wire, and has a mechanism that can mechanically change the position and posture of the camera by controlling the driving force of a motor or the like with a control signal. be. The mobile camera unit 205 can physically change the position and orientation of the camera according to instructions from the camera mechanism control unit 206, which will be described later.

The camera mechanism control unit 206 transmits a control signal to the mobile camera unit 205 in accordance with an input from the user according to the operation of the user operation unit 204, and controls the mobile camera unit 205 to physically change its position and posture. I do. At this time, the camera mechanism control unit 206 can perform control based on physical information such as up/down/left/right, forward/backward, and rotation, as well as designation using world coordinates and control using camera parameters. The camera mechanism control unit 206 can also transmit current camera parameters to the camera parameter calculation unit 207, which will be described later.

A camera parameter calculation unit 207 calculates camera parameters of the mobile camera unit 205 . Camera parameters are calculated so as to track the subject specified by the user operation unit 204 from the subjects extracted by the subject extraction unit 203 . Also, when the user operates the user operation unit 204, the result of the user operation can be output as a camera parameter. At this time, for example, if it is determined that the obtained camera parameters are not separated from another subject extracted by the subject extraction unit 203 by a predetermined distance or more and there is a risk of collision, the above-described camera parameters are changed. Redo the calculation.

The camera movable range acquisition unit 208 provides the camera parameter calculation unit 207 with information about the operable range of the mobile camera unit 205 controlled by the camera mechanism control unit 206 . The operable range indicates the movable range of the camera, and is the range in which the mobile camera section 205 can actually capture the target subject as an image using the camera mechanism control section 206 .

As a result, the camera parameter calculation unit 207 determines a range within which the mobile camera unit 205 can be physically moved by the camera mechanism control unit 206 and within which the mobile camera unit 205 can focus by zooming or focusing the camera. The camera parameters can be calculated in

A virtual viewpoint image generation unit 209 generates a virtual viewpoint image using the virtual viewpoint image data of the shape estimation processing unit 202 . The virtual viewpoint image generation unit 209 also receives a user operation of a virtual viewpoint in the virtual viewpoint image from the user operation unit 204 .

The imaging control system 100 according to this embodiment simultaneously captures a virtual viewpoint image with a physical camera, that is, a camera fixedly installed in a photography facility such as a stadium, or while moving in the space within the photography facility. This is applicable when shooting with a wire camera, crane camera, or the like. For example, it is conceivable to broadcast a sport such as soccer or shoot an event such as a concert. In such filming, athletes and people performing are filmed using a physical camera, and at the same time, free-viewpoint video is used to capture images that are difficult to capture with a physical camera, such as athletes during a match. It is possible to shoot with a sense of realism, such as right next to the. In such photography, the camera is operated not only by the camera operator, but also by the operation of automatically tracking the ball or the player. explain. It should be noted that the present invention is not limited to this, and any type of photography, such as event photography such as other sports or concerts, may be used without departing from the scope of the embodiments.

In addition, there are various methods for obtaining the position and coordinates of the physical camera in such photographing. By performing accurate mechanical control, the coordinates may be obtained from the difference value from the reference position, or the information may be read by a sensor using a special marker or the like. Alternatively, the physical shape of the camera may be obtained from the free-viewpoint video, and the coordinate information may be obtained therefrom, or any method may be used.

<Processing>
FIG. 3 is a flowchart showing the procedure of processing when the mobile camera unit 205 tracks a subject in the shooting control system 100 according to this embodiment.

In step S301, the shape estimation processing unit 202 executes shape estimation processing using image data captured by the virtual viewpoint image generation camera unit 201, and generates data necessary for generating a virtual viewpoint image. Any method may be used as a method for generating data necessary for generating a virtual viewpoint image. For example, a visual volume intersection method (Visual Hull) may be used. Using the three-dimensional shape data generated in step 3 and information such as color information and bounding box, the subject is extracted. Here, for example, we are shooting a soccer match, and in addition to soccer balls, individual players, referees, etc., the physical shooting positions of crane cameras, wire cameras, drone cameras, etc. installed near or above the field The shape of the camera capable of changing is estimated by the shape estimation processing unit 202 . Based on the result of the shape estimation, the subject extraction unit 203 performs extraction processing as an independent individual subject.

In step S303, the user operation unit 204 accepts designation of an arbitrary subject among the independent subjects extracted in step S302. For example, a specification of a soccer ball is accepted here. In the present embodiment, it is assumed that the soccer ball has been designated by the user, but the present invention is not limited to this. Any subject designation may be accepted as long as it is extracted as an independent subject in step S302. It is possible to accept the designation of various subjects whose shapes are estimated and extracted as independent subjects, such as people such as athletes and referees, crane cameras, wire cameras, and drone cameras. In addition, when the subject has already been specified by the user, the same subject previously specified by the user may be specified unless the user again specifies another subject.

Here, FIG. 4 is a schematic diagram schematically showing how a subject is specified according to the present embodiment. Graphics display 400 represents a subject designation screen displayed on a user's display device (not shown). The virtual viewpoint image generated by the shape estimation processing unit 202 is displayed on the screen so that the entire virtual viewpoint image can be viewed from above, and subjects such as players and balls are displayed in the virtual viewpoint image.

Further, bounding boxes are displayed for these subjects, and the user designates, for example, a player 401 or a ball 402 from among these using a pointing device such as a mouse or a touch panel, which is the user operation unit 204 . Here, it is assumed that the ball 402 has been selected by the user. Note that the subject selection method is not limited to this method. For example, a number or the like may be displayed in each bounding box, and selection may be made by specifying those numbers by key input.

In step S<b>304 , the camera parameter calculation unit 207 obtains camera parameters that allow the subject to be captured within the camera within the movable range acquired by the camera movable range acquisition unit 208 with respect to the world coordinates of the subject specified in step S<b>303 . is obtained by calculation. Then, the obtained camera parameters are sent to the camera mechanism control unit 206 .

Here, FIGS. 5A to 5C are schematic diagrams schematically showing how a subject is tracked by the wire camera 502 according to this embodiment. A wire 501 is controlled by an instruction from the camera mechanism control unit 206 and performs operations such as physical movement and rotation of a wire camera 502 which is the mobile camera unit 205 . The wire camera 502 can be freely manipulated three-dimensionally on the field by wires connected to the four corners of the stadium.

FIG. 5(a) shows how a player 401 kicks a ball 402, traces the trajectory of an arrow 503, and then moves to a post-movement ball position 504. FIG. FIG. 5B shows an image captured by the wire camera 502 at this time. In this example, it is assumed that tracking is performed so that the ball 402 to be tracked is within a certain range at the center of the screen. A ball 402 kicked by a player 401 moves rapidly along the trajectory of an arrow 503, and the ball 402 to be tracked has moved outside the angle of view. At this time, the wire camera 502 alone cannot determine the position of the ball and cannot continue tracking. However, it is possible to extract the ball at the ball position 504 after movement based on the information from the subject extraction unit 203 . Therefore, the camera parameter calculation unit 207 calculates a camera position 505 shown in FIG. 5C as the movement destination position of the wire camera 502 for tracking the ball according to the movement of the ball. By obtaining the camera parameters at this time and sending the obtained camera parameters to the camera mechanism control unit 206, it is possible to track the subject that has moved outside the angle of view.

In this example, the position of the wire camera 502 is moved to obtain the destination position of the wire camera 502, but any method may be used. For example, the wire camera 502 may calculate the amount of rotational movement and control the rotation so that the ball position 504 after movement can be captured by rotational movement such as panning or tilting without moving from that position. Alternatively, the ball may be tracked by panning or tilting, and the wire 501 may be operated to move the position of the wire camera 502 by user operation via the user operation unit 204, or an operation such as zooming may be separately performed. good. Alternatively, the camera mechanism control unit 206 may control only the operation of moving the position of the wire camera 502 by the user, such as panning and tilting, based on the camera parameters obtained by the camera parameter calculation unit 207. .

In step S305, the camera parameter calculation unit 207 predicts the motion of the subject based on the motion amount of the model information generated by the shape estimation processing unit 202 in step S301 (calculates and acquires information about the predicted motion of the subject). . Then, based on the camera parameters obtained in step S304, it is determined whether or not there is a risk of collision between the mobile camera unit 205 and subjects including subjects other than the tracking target, such as players, in addition to the ball that is the tracking target. do. If it is determined that there is a risk of collision with the subject, the process returns to step S304 to obtain the camera parameters again. On the other hand, if it is determined that there is no risk of collision with the object, the process proceeds to step S306.

Here, FIGS. 6A and 6B are diagrams showing how the camera parameter calculation unit 207 obtains camera parameters when a collision is predicted by calculation of camera parameters according to the present embodiment. .

As in FIGS. 5A to 5C, when the wire camera 502 is to follow the ball position 504 after movement and is to be moved to the camera position 505, the player 601 moves along the movement line of the wire camera 502. At the same time, player 602 is on camera position 505 . Therefore, moving the wire camera 502 to the camera position 505 may collide with the players 601 and 602 . Therefore, the camera parameter calculation unit 207 obtains the camera parameters again so as to move to a position other than the camera position 505, and obtains a camera position 603 as shown in FIG. 6B, for example. At this time, for the camera position 505, for example, the camera parameters may be obtained so as to track the tracking target at a certain distance. Then, for the camera position 603 , the camera parameters may be determined so that the ball can be seen in the same size as the camera position 505 by performing zooming by the distance. In this way, priority may be given to each parameter when obtaining camera parameters, and camera parameters may be obtained according to the priority.

In step S306, the camera mechanism control unit 206 controls tracking of the subject by moving the mobile camera unit 205 (changing the position and orientation) based on the camera parameters received from the camera parameter calculation unit 207 in step S304. conduct.

In step S307, the user operation unit 204 determines whether or not to end the process. If the user does not give an instruction to end processing, the process returns to step S701.

As described above, according to the present embodiment, by using the data for generating the virtual viewpoint image (the model information of the subject), the subject can be continuously tracked without losing sight of the subject. Also, collision with the subject can be avoided.

In this embodiment, the case where the subject to be tracked has moved outside the screen (outside the angle of view) has been described, but the present invention is not limited to this. For example, when the camera parameters are determined in step S304, the ball may be hidden by many players in front of the camera, causing occlusion. In such a case, a position where occlusion does not occur may be calculated from the position information of the ball so that the subject to be tracked is not hidden.

In this case, the camera mechanism control unit 206 controls the mechanism based on the model information generated by the shape estimation processing unit 202 so that the subject to be tracked is not hidden behind other subjects in the image captured by the mobile camera unit 205. .

(Embodiment 2)
In this embodiment, by using the data for generating the virtual viewpoint image, the subject can be photographed by preparing for photographing the subject before the camera can capture the subject. An example will be described in which shooting is immediately performed when a situation arises.

<Configuration>
Since the hardware configuration of the imaging control system according to this embodiment is the same as the hardware configuration shown in FIG. 1, the description thereof will be omitted.

FIG. 7 shows an example of functional blocks of an imaging control system 700 according to this embodiment. Note that the virtual viewpoint image generation camera unit 201 to the virtual viewpoint image generation unit 209 shown in FIG. 7 are the same as the components in FIG.

The subject photographable range acquisition unit 701 provides the camera parameter calculation unit 207 with information on the operable range in which the mobile camera unit 205 is controlled by the camera mechanism control unit 206 .

<Processing>
Here, FIG. 8 is a flowchart showing the procedure of processing when the mobile camera unit 205 tracks the subject in the imaging control system 700 according to this embodiment. Note that steps S301 to S307 in FIG. 8 are the same as the corresponding steps in FIG. 3, so descriptions thereof will be omitted.

In step S801, the camera parameter calculation unit 207 acquires information on the operable range of the mobile camera unit 205 from the subject photographable range acquisition unit 701, and the subject specified in step S303 is within the operable range of the mobile camera unit 205. It is determined whether or not there is If not within the operable range, the process proceeds to step S802. On the other hand, if it is within the operable range, the process proceeds to step S304.

In step S<b>802 , the camera parameter calculation unit 207 acquires the position information of the tracking target subject, and obtains camera parameters that are closest to the tracking target subject within the operable range of the mobile camera unit 205 .

Here, FIGS. 9A and 9B are schematic diagrams showing how the crane camera according to the present embodiment tracks the ball.

As shown in FIG. 9A, a crane 901 is controlled by instructions from the camera mechanism control unit 206, and performs operations such as physical movement and rotation of a crane camera 902, which is the mobile camera unit 205. FIG.

A operable range 903 indicates an area that can be photographed by the crane camera 902 under the control of the crane 901 . The crane camera 902 can track the subject inside the operable range 903 and cannot track the subject outside the operable range 903 . Assume here that the crane camera 902 is set to track the ball 402 .

The ball 402 is within the operable range 903 of the mobile camera unit 205 and is being tracked by the mobile camera unit 205 .

Assume that the player 401 kicks the ball 402 from this state and moves to the ball position 904 outside the movable range 903 . Since the ball position 904 is outside the operable range 903 , it cannot be tracked by the crane camera 902 . At this time, the camera parameter calculation unit 207 obtains camera parameters for an angle of view 905 that is considered to be the closest position to the ball position 904 within the area of the operable range 903, and directs the crane camera 902 to the angle of view 905. Let In this operation, every time the position of the ball position 904 moves outside the operable range 903, the angle of view of the angle of view 905 may be appropriately corrected by recalculating the camera parameters according to the position of the ball. .

For example, as shown in FIG. 9B, the ball 402 to be tracked changes its position from a ball position 904 to a ball position 904'' and into the movable range 903. To go. In response to this, the crane camera 902 changes the angle of view according to the position of the ball 402 from the angle of view 905 through the angle of view 905 ′ to the angle of view 905 ″ by the camera parameter calculator 207 . As a result, at a ball position 904'' where the ball 402 has entered the movable range 903, the ball can be immediately caught by the angle of view 905''.

As described above, according to the present embodiment, by using data for generating a virtual viewpoint image, preparations for photographing a subject can be made before a camera such as a crane camera can capture the subject. I do. Therefore, when the subject can be captured, the subject can be photographed immediately.

It should be noted that although the operable range 903 has only one continuous region in the present embodiment, it is not limited to this. In the case of a mobile camera with multiple cameras such as a compound eye camera, it may have multiple operable ranges, and it may have a mechanism that physically changes the lens, weather or other restrictions. A plurality of operable ranges may be switched according to the state of the camera.

(Embodiment 3)
In this embodiment, an example will be described in which an appropriate camera is used to track a subject by appropriately switching the camera for shooting according to the position of the tracking target.

<Configuration>
Since the hardware configuration of the imaging control system according to this embodiment is the same as the hardware configuration shown in FIG. 1, the description thereof will be omitted.

FIG. 10 shows an example of functional blocks of an imaging control system 1000 according to this embodiment. Note that the virtual viewpoint image generation camera unit 201 to the virtual viewpoint image generation unit 209 shown in FIG. 10 are the same as the components in FIG. However, in the example of FIG. 10, a mobile camera section 205′ is provided in place of the mobile camera section 205, and the mobile camera section 205′ is composed of at least two or more mobile camera sections 205. FIG. A camera mechanism control unit 206 ′ is provided in place of the camera mechanism control unit 206 , and the camera mechanism control unit 206 ′ is composed of at least two or more camera mechanism control units 206 . A subject shootable range acquisition unit 701 is the same as the components in FIG.

The switching unit 1001 uses the camera parameter calculation unit 207 to determine whether or not the subject to be tracked is within the photographable range of each mobile camera unit 205′. Then, an instruction is issued to an external video switching device through communication via the external I/F 104 so as to switch to the mobile camera unit 205 capable of shooting.

Note that in the present embodiment, switching is realized by issuing a communication instruction to an external switching device, but the present invention is not limited to this. You may Alternatively, it may be processed by software by an arithmetic device such as the CPU 101 or the GPU 105 without adding a new device.

<Processing>
Here, in FIG. 11, in the shooting control system 1000 according to the present embodiment, when a plurality of cameras are used to track an object using the mobile camera unit 205′, tracking is performed while switching to an appropriate camera. 10 is a flow chart showing a procedure of processing at the time; Note that steps S301 to S307 in FIG. 11 are the same as the corresponding steps in FIG. 3, and steps S801 and S802 are the same as the corresponding steps in FIG.

In step S1101, the camera parameter calculation unit 207 determines whether or not the tracking operation processing of the mobile camera units 205' has been completed for all of the mobile camera units 205'. If all tracking operation processing has been completed, the process advances to step S1102. On the other hand, if all tracking operation processing has not ended, the process returns to step S801.

In step S<b>1102 , the camera parameter calculation unit 207 determines that the object being tracked is within the photographable range of the currently selected mobile camera unit 205 based on the information acquired by the subject photographable range acquisition unit 701 . Determine whether or not If the subject being tracked is within the photographable range of the currently selected mobile camera unit 205, the process proceeds to step S307. On the other hand, if the subject being tracked is outside the photographable range of the currently selected mobile camera unit 205, the process advances to step S1103.

In step S1103, the camera parameter calculation unit 207 obtains the mobile camera 205 in which the tracking subject is captured within the photographable range. Then, the switching unit 1001 switches to the mobile camera 205 in which the tracked subject is captured within the photographable range.

Here, FIG. 12 is a schematic diagram showing a state of a soccer match according to this embodiment. Switching of cameras will be described with reference to FIG.

In FIG. 12, a crane 1201a is controlled by instructions from the camera mechanism control section. The crane camera 1202a is a mobile camera for which the photographable range 1203a is set as the photographable range. Similarly, crane 1201b is controlled by instructions from the camera mechanism controller. The crane camera 1202b is a mobile camera in which the photographable range 1203b is set as the photographable range. The crane 1201c is controlled by instructions from the camera mechanism controller. The crane camera 1202c is a mobile camera in which the photographable range 1203c is set as the photographable range.

A description will be given of the operation when the ball, which is the tracking target, is photographed using such a mobile camera group and the ball moves to positions 1204a, 1204b, and 1204c.

When the ball is at the position 1204a, the camera parameter calculation unit 207 instructs the switching unit 1001 to switch to the crane camera 1202a because the ball is within the photographable range 1203a. The crane camera 1202a controls the crane camera 1202a by the camera parameter calculation unit 207 so as to follow the ball to be tracked through the processing of steps S301 to S306. At this time, the crane camera 1202b and the crane camera 1202c are controlled by the processing in step S802 so as to be closest to the ball within each photographable range.

Next, when the ball moves to position 1204b, the camera parameter calculation unit 207 determines that the position of the ball moves outside the photographable range 1203a and the ball exists within the photographable range 1203b. Therefore, the switching unit 1001 is instructed to switch to the crane camera 1202b. At this time, the crane camera 1202b is controlled by the processing in step S802 so as to be closest to the ball, so tracking can be started immediately after the ball crosses the boundary 1205 of the photographable range 1203b. Therefore, when switching from the crane camera 1202a to the crane camera 1202b, it is possible to switch while tracking the ball. Also, after the crane camera 1202a cannot track the ball, it is controlled by the processing in step S802 so as to come closest to the ball within the photographable range 1203a.

After that, similarly, when the ball moves to position 1204c, an instruction is issued to switching unit 1001 to switch from crane camera 1202b to crane camera 1202c.

In addition, in the area where the shooting range overlaps, the currently used camera may be used as described above. You may switch to the camera of the area|region of a move-destination with which possible range overlaps. Also, the two images may be switched by a transition.

As described above, according to the present embodiment, the camera can be appropriately switched (switched) according to the position of the tracking target (for example, the position of the ball), thereby tracking the subject using an appropriate camera. can.

(Embodiment 4)
In the present embodiment, an example will be described in which switching to a camera that cannot capture a subject to be tracked is suppressed, thereby suppressing display of an image in which the subject is not captured.

<Configuration>
Since the hardware configuration of the imaging control system according to this embodiment is the same as the hardware configuration shown in FIG. 1, the description thereof will be omitted. Also, since the configuration of the functional blocks of the imaging control system according to this embodiment is the same as the configuration shown in FIG. 10, description thereof will be omitted.

<Processing>
FIG. 13 is a flowchart showing the procedure of processing according to this embodiment. This flow represents switching processing for the mobile camera unit 205' based on user operation or based on reception of a signal from an external device. 13 are the same as the corresponding processes described with reference to FIGS. 3 and 8. In FIG.

In step S1301, the user operation unit 204 receives input of a switching instruction based on user operation. Alternatively, the switching instruction may be received based on the reception of the switching signal from the external device.

In the present embodiment, determination processing in step S801 after step S1301 is performed, and depending on the determination result as to whether or not the subject is within the operable range, the process proceeds to step S1302 if Yes, and to step S1303 if No. , the process branches.

In step S1302, the camera parameter calculation unit 207 accepts the switching instruction and instructs the switching unit 1001 to switch the video because the subject to be tracked is within the operable range.

In step S1303, the camera parameter calculation unit 207 does not accept a switching instruction because the subject to be tracked is outside the operable range. At this time, a graphic display, a text message, or the like indicating that the switching instruction is not accepted may be displayed on the display device.

In this manner, when an instruction to switch the mobile camera unit is input through the user operation unit 204, switching of the mobile camera unit is performed depending on whether or not the subject to be tracked is within the movable range of the mobile camera unit to which switching is to be performed. control whether or not

Here, FIG. 14 is a schematic diagram schematically showing a state of a soccer match according to this embodiment. The behavior when the camera switches will be described with reference to FIG.

The position and orientation of the crane camera 1402 are three-dimensionally changed by the crane 1401 . Crane camera 1402 is a camera capable of capturing an image within photographable range 1403, and crane camera 1402 is set to track a soccer ball.

If an instruction to switch to the crane camera 1402 is received by the user's operation when the soccer ball is at the position 1404a, the switching process is accepted because the soccer ball is within the photographable range. Then, switching processing to the image of the crane camera 1402 is performed.

On the other hand, when the soccer ball is at the position 1404b, if an instruction to switch to the crane camera 1402 is received by the user's operation, the soccer ball is outside the photographable range, so the crane camera 1402 cannot photograph the soccer ball. be. In this case, switching processing is not accepted. In this embodiment, it is assumed that the switching process is not accepted, but the present invention is not limited to this. Although the switching process is accepted, a display may be made to notify the user that the subject cannot be tracked. Alternatively, the virtual viewpoint image may be generated and output by the virtual viewpoint image generation unit 209 after notifying those users. At that time, information indicating that switching to the virtual viewpoint image generated by the virtual viewpoint image generation unit 209 has been performed may be displayed to the user because the switching was not possible.

Alternatively, if it is outside the photographable range of the mobile camera unit 205' specified by the user but is within the photographable range of another mobile camera unit 205', the subject to be tracked is within the photographable range. may be switched to the mobile camera unit 205' located at .

Also, in the present embodiment, the case where there is one mobile camera unit 205' (the crane camera 1402) is shown, but the present invention is not limited to this. Even if there are a plurality of mobile camera units 205', processing can be performed for one camera for which a switching instruction is given by the user.

As described above, according to the present embodiment, switching to a camera that cannot capture the subject to be tracked is suppressed, and display of an image that does not capture the subject to be tracked is suppressed. can do.

(Embodiment 5)
In this embodiment, an example of avoiding collision with a subject that is not captured by the camera when tracking the subject will be described.

<Configuration>
Since the hardware configuration of the imaging control system according to this embodiment is the same as the hardware configuration shown in FIG. 1, the description thereof will be omitted. Also, since the configuration of the functional blocks of the imaging control system according to this embodiment is the same as the configuration shown in FIG. 10, the description thereof will be omitted.

<Processing>
FIG. 15 is a flowchart showing the procedure of processing according to this embodiment. Note that each process of steps S301 to S307, steps S801 and S802, and steps S1101 to S1103 is the same as the corresponding processes in FIGS.

In this embodiment, if the result of the collision determination process with the subject in step S305 is Yes, the process proceeds to step S1501, and if No, the process proceeds to step S306.

In step S1501, the camera parameter calculation unit 207 obtains camera parameters of the mobile camera unit that avoid the collision because the collision with the object is predicted in step S305. That is, the camera parameters for retracting the mobile camera unit are obtained.

In step S1502, the switching unit 1001 performs switching processing from the mobile camera unit (eg, crane camera) that performed the avoidance processing in step S1501 to another mobile camera unit (eg, wire camera).

Here, FIG. 16 is a schematic diagram showing part of a scene of a soccer match according to this embodiment. FIG. 16 shows how a player 1601 kicks the ball over the head of a player 1602 and flies along a trajectory 1603 .

A crane camera 1604 is a camera preset by the user to track the goalkeeper 1605 .

The wire camera 1606 is a camera that is set in advance by the user to track the goalkeeper 1605 and shoots, like the crane camera 1604 .

The ball draws a trajectory 1603 and is approaching the crane camera 1604. The camera parameter calculation unit 207 determines that a collision is predicted by the processing in step S305, and the camera parameters for avoiding the collision are obtained in step S1501. . For example, camera parameters for crane camera 1604 to stop tracking goalkeeper 1605 and to retreat to position 1607 are obtained. Since the crane camera 1604 is retracted, the image is switched to the image of the wire camera 1606 by the switching unit 1001 in the process of step S1502. Then, the video image of the goalkeeper 1605 captured by the wire camera 1606 can be continuously output.

As described above, according to the present embodiment, when tracking a subject, a subject that is not captured in the image of a certain mobile camera section (the crane camera 1604 in the example of FIG. 16) and the mobile camera section can be detected. Collisions can be avoided. Furthermore, it is possible to prevent a situation in which a desired image cannot be captured due to avoidance.

In this embodiment, switching to another mobile camera unit is performed at the same time as avoidance, but the present invention is not limited to this. For example, while the crane camera 1604 is avoiding, if the goalkeeper 1605 to be tracked can be tracked by the crane camera 1604 by panning, tilting, or the like, the tracking may be continued. Then, when tracking becomes impossible, control may be performed so as to switch to another mobile camera unit.

Also, even if the wire camera 1606 switched by the switching process in step S1502 is tracking the ball instead of the goalkeeper 1605, the switching may be performed. Also, the crane camera 1604 may change control so as to newly track the goalkeeper 1605 at the same time as the switching.

In addition, when the crane camera 1604 that has retreated to the position 1607 finishes avoiding the ball and starts tracking the goalkeeper 1605 again, and it becomes possible to photograph the goalkeeper 1605, the image from the wire camera 1606 is sent to the crane camera 1604. may be controlled to switch.

Also, the video to be switched when collision avoidance with a subject is performed may be a virtual viewpoint image of a virtual camera instead of an image of a specific mobile camera unit. In addition, the camera parameters of the virtual viewpoint image at this time may continue video generation as camera parameters that can be taken when a collision is not predicted based on the camera parameters of the crane camera 1604 before avoidance. After the crane camera 1604 finishes the collision avoidance processing and starts tracking the goalkeeper 1605 again, it may be controlled to switch to the image of the crane camera 1604 . Also, the image switching from the virtual camera to the crane camera 1604 at this time may be performed when the camera parameters of the crane camera 1604 and the camera parameters of the virtual camera match.

Further, when switching to the virtual viewpoint image of the virtual camera, it may be the case that the collision with the subject is predicted for all the mobile camera units. Also, when switching to the virtual viewpoint image of the virtual camera, it may be the case that there is no other mobile camera unit in which the subject to be tracked is captured in the captured image.

[Modified example of each embodiment]
In each of the above-described embodiments, a soccer match is described as an example, but the present invention is not limited to this. Any object such as sports other than soccer, a concert stage, a play, a lecture, or the like may be photographed.

Further, in each of the above-described embodiments, the operable range is a range in which the mobile camera unit 205 can actually capture an image of the target subject using the camera mechanism control unit 206. , but not limited to. The operable range may be a range that satisfies a certain condition when the target subject is photographed by the mobile camera unit 205 . A certain condition may indicate a case in which the target subject can be captured within a reference range of the image, such as within m pixels from the center of the pixels of the image captured by the mobile camera unit 205, for example. . Alternatively, the threshold may be determined according to the resolution, such as a range in which the target object can be photographed with a resolution of n pixels or more per meter with respect to the size of the actual object. In addition, the reference of the resolution may be based on the bounding box of the object instead of directly using the shape of the subject. Also, the operable range may be a coordinate range designated by the user.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: shooting control system, 201: virtual viewpoint image generation camera unit, 202: shape estimation processing unit, 203: subject extraction unit, 204: user operation unit, 205: mobile camera unit, 206: camera mechanism control unit, 207: Camera parameter calculation unit, 208: camera movable range acquisition unit, 209: virtual viewpoint image generation unit

Claims (13)

a shape estimating means for estimating a three-dimensional shape of a subject from a plurality of captured images captured by a plurality of imaging devices and generating model information of the subject;
virtual viewpoint image generation means for generating a virtual viewpoint image based on the plurality of captured images and the model information;
control means for controlling to change the position and orientation of at least one photographing means different from the plurality of photographing devices based on the model information;
A shooting control system comprising: further comprising user operation means for accepting designation of a subject based on a user operation from among the subjects whose three-dimensional shape has been estimated,
2. The photographing control system according to claim 1, wherein said control means controls said photographing means so as to track a subject designated by said user's operation. 3. The apparatus according to claim 2, wherein said control means controls said photographing means so as to track the subject designated by said user operation within a movable range of the position and orientation of said at least one photographing means. shooting control system. When the subject to be tracked is not within the movable range of the position and orientation of the at least one photographing means, the control means controls the at least one photographing means to be closest to the subject to be tracked within the movable range. 4. The photographing control system according to claim 3, wherein the photographing means is controlled. 3. The controlling means controls the photographing means so that the subject to be tracked is not hidden behind other subjects in the photographed image of the at least one photographing means, based on the model information. 5. The imaging control system according to any one of 4. Determination means for determining whether or not a subject to be tracked is shown in an image captured by the first imaging means;
a switching means for switching to a second photographing means in which the subject to be tracked is shown when the judging means judges that the subject to be tracked is not shown in the photographed image of the first photographing means;
6. The imaging control system according to any one of claims 1 to 5, further comprising: When an instruction to switch the photographing means is input by the user operation means, a control means for controlling whether or not the photographing means can be switched according to whether or not the subject to be tracked is within the movable range of the photographing means to be switched to. 5. The imaging control system according to any one of claims 2 to 4, further comprising: 8. The photographing control system according to claim 7, wherein the control means does not switch the photographing means when the subject to be tracked is not within the movable range of the photographing means to be switched to. further comprising: first photographing means for photographing a subject to be tracked based on the model information; and prediction means for predicting a collision with at least one of the subject to be tracked and another subject;
6. The photographing control system according to claim 1, wherein when the collision is predicted, the control means retracts the first photographing means so as to avoid the collision. . 10. The photographing control system according to claim 9, further comprising switching means for switching from said first photographing means retracted by said control means to second photographing means. 10. The photographing according to claim 9, further comprising switching means for switching from the photographed image of said first photographing means saved by said control means to the virtual viewpoint image generated by said virtual viewpoint image generating means. control system. The switching means is operated when a collision with the subject is predicted for all other photographing means except for the first photographing means, or when there is another photographing means in which the subject to be tracked appears in the photographed image. 12. The photographing control system according to claim 11, wherein the image photographed by the first photographing means is switched to the virtual viewpoint image when the image is not to be displayed. A method of operating a capture control system, comprising:
a shape estimation step of estimating a three-dimensional shape of a subject from a plurality of captured images captured by a plurality of imaging devices and generating model information of the subject;
a virtual viewpoint image generating step of generating a virtual viewpoint image based on the plurality of captured images and the model information;
a control step of controlling to change the position and orientation of at least one photographing device different from the plurality of photographing devices based on the model information;
A method of operating a photographing control system, comprising:

Images