JP6621063B2 - Camera selection method and video distribution system - Google Patents

Description

  The present invention relates to a camera selection method for selecting a camera from which a video to be displayed is shot from a plurality of cameras shooting the same scene.

  As a video distribution method, Patent Literature 1 discloses a technique for controlling shooting conditions of a camera in a multi-view video distribution system. Patent Document 2 discloses a technique for distributing videos taken from a plurality of viewpoints in conjunction with viewpoint movement.

JP 2002-165200 A JP 2012-094990 A

  However, the above technique does not describe how to select a camera from many cameras when there are many cameras.

  Therefore, an object of the present invention is to provide a camera selection method or a video distribution system that can appropriately select a camera from a plurality of cameras.

In order to achieve the above object, a camera selection method according to an aspect of the present invention provides a camera (M (M) of a video to be displayed) from N (two or more natural numbers) cameras that are shooting the same scene. is a camera selection method for selecting a natural number smaller than N) cameras, circuit, an acquisition step of acquiring the position and photographing direction of the N cameras, wherein the circuit is obtained for the N number based on the position and the shooting direction of the camera, viewing contains a selection step of selecting the M cameras from the N cameras, wherein the selection step, seen along the conditions i) the N cameras in a vertical direction The M cameras surround a predetermined position, and condition ii) each of the M cameras has a shooting direction for shooting the predetermined position, and condition iii) the N cameras Camera in the vertical direction The angle formed by the shooting directions of the two adjacent cameras included in the M cameras is the shooting direction of the other two adjacent cameras included in the M cameras. The M cameras satisfying the same size as the angle formed by are selected .

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. The system, method, integrated circuit, computer program Also, any combination of recording media may be realized.

  The present invention can provide a camera selection method or a video distribution system that can appropriately select a camera from a plurality of cameras.

It is a figure which shows the structure of the video delivery system which concerns on embodiment. It is a block diagram of a server concerning an embodiment. It is a flowchart of the camera information calculation process which concerns on embodiment. It is a figure for demonstrating the three-dimensional reconstruction which concerns on embodiment. It is a flowchart of the image | video selection process which concerns on embodiment. It is a flowchart of the initial camera selection process which concerns on embodiment. It is a flowchart of the camera switching determination process which concerns on embodiment. It is a flowchart of the tracking camera switching determination process which concerns on embodiment. It is a flowchart of the switching camera selection process which concerns on embodiment. It is a figure for demonstrating the operation | movement which tracks a to-be-photographed object using the two-dimensional image which concerns on embodiment. It is a figure for demonstrating the operation | movement which tracks a to-be-photographed object using the two-dimensional image which concerns on embodiment. It is a figure for demonstrating the operation | movement which tracks a to-be-photographed object using the three-dimensional model which concerns on embodiment. It is a figure for demonstrating the operation | movement which tracks a to-be-photographed object using the two-dimensional image and three-dimensional model which concern on embodiment. It is a flowchart of the reset determination process which concerns on embodiment. It is a figure for demonstrating the camera selection in the space imaging | photography mode in an event venue based on embodiment. It is a figure for demonstrating the camera selection in the space imaging | photography mode in an event venue based on embodiment. It is a figure for demonstrating the camera selection in the tracking imaging | photography mode in an event venue based on Embodiment. It is a figure for demonstrating the camera selection in the tracking imaging | photography mode in an event venue based on Embodiment. It is a figure for demonstrating the camera selection in the tracking imaging | photography mode in an event venue based on Embodiment. It is a figure for demonstrating the camera selection in the space imaging mode in the monitoring environment based on Embodiment. It is a figure for demonstrating the camera selection in the tracking imaging | photography mode in the monitoring environment based on Embodiment. It is a flowchart of the initial camera selection process which concerns on embodiment. It is a flowchart of the evaluation value calculation process of the camera unit according to the embodiment. It is a figure which shows the example of selection of the target area | region which concerns on embodiment. It is a figure which shows the example of selection of the to-be-photographed object which concerns on embodiment. It is a flowchart of the camera position evaluation value calculation process which concerns on embodiment. It is a figure which shows the camera selection example in the bird's-eye view mode which concerns on embodiment. It is a figure which shows the example of camera selection in the concentration mode which concerns on embodiment. It is a flowchart of the camera selection method which concerns on embodiment.

(Knowledge that became the basis of the present invention)
As a video distribution system, for example, a system in which videos taken by a plurality of cameras (for example, smartphones) owned by a plurality of users are stored on a server and distributed to viewers is assumed. In such a system, the number of videos held by the server is enormous, but the number of videos that can be displayed by the terminal device owned by the viewer is limited.

  In such a case, it is difficult for the viewer to check all the videos and select a desired video from them. Further, since the types (functions) of a plurality of cameras owned by a plurality of users are assumed to be different from each other, it is necessary to cope with this.

  The camera selection method according to an aspect of the present invention includes M (M is a natural number greater than or equal to 2) cameras (M is a natural number smaller than N) from which N images are captured from N (2 or more natural numbers) cameras capturing the same scene. A camera selection method for selecting a camera, the acquisition step of acquiring the positions and shooting directions of the N cameras, and the N cameras based on the acquired positions and shooting directions of the N cameras. Selecting the M cameras.

  Thereby, the camera selection method can appropriately select a camera from a plurality of cameras based on the position and shooting direction of the camera.

  For example, in the selection step, there is less overlap of the shooting spaces of the M cameras based on the positions and shooting directions of the N cameras, and the shooting space of the M cameras covers the target space. The larger M cameras may be selected with priority.

  Thereby, the camera selection method can select a plurality of cameras so that the entire target area can be covered by the plurality of cameras.

  For example, the selection step may preferentially select M cameras that are shooting the target space or the target subject at equal intervals from a plurality of directions based on the positions and shooting directions of the N cameras. .

  Thereby, the camera selection method can select a plurality of cameras so that the images of the target subject in all directions can be covered by the plurality of cameras.

  For example, in the obtaining step, the positions and photographing directions of the N cameras may be calculated by performing three-dimensional reconstruction using images photographed by the N cameras.

  Thereby, the imaging range of each camera can be determined with high accuracy.

  For example, in the acquisition step, the position and shooting direction of the N cameras may be calculated using information obtained from a sensor included in the camera and transmitted from each of the N cameras.

  Thereby, the position and shooting direction of each camera can be easily determined.

  For example, in the obtaining step, focal lengths of the N cameras are further obtained, and in the selecting step, the N cameras are based on the obtained positions, shooting directions, and focal lengths of the N cameras. May select the M cameras.

  Thereby, the imaging range of each camera can be determined with high accuracy.

  For example, the camera selection method further includes a first determination for determining whether or not to switch a part of the selected M cameras to another camera in a frame after the selection step. And a switching step of selecting a new camera instead of the part of the cameras based on the positions and shooting directions of the N cameras when it is determined to switch in the first determination step. But you can.

  Thereby, compared with the case where all the cameras are switched, processing amount can be reduced.

  For example, in the first determination step, the target subject may be tracked between frames, and if the tracking of the target subject fails, it may be determined that the selected camera is switched.

  For example, in the first determination step, a three-dimensional model of the target subject is associated between frames, the associated three-dimensional model of the target subject is projected onto a frame at the current time, and the target subject is included in the obtained frame. May not be present, it may be determined that the tracking has failed.

  As a result, the subject can be tracked with high accuracy.

  For example, when the camera selection method further performs the selection step in a second determination step for determining whether or not to perform the selection step again in a frame after the selection step, and the second determination step. If it is determined, a re-selecting step of reselecting the M cameras from the N cameras based on the positions and shooting directions of the N cameras may be included.

  For example, in the first determination step, when the elapsed time from the previous switching is less than the first time, it is determined that the switching is not performed, and the elapsed time is longer than the first time and longer than the first time. If it is less than the second time, it is determined by the first reference whether or not to switch the part of the cameras to the other camera, and if the elapsed time is not less than the second time, the part of the cameras is Whether to switch to the other camera may be determined based on a second reference that is looser than the first reference.

  Thereby, it can suppress that the selected camera switches frequently.

  For example, the selecting step is based on a step of calculating a first evaluation value of each of the N cameras based on an image captured by the N cameras, and based on a position and a shooting direction of the N cameras. Calculating a second evaluation value of each camera group that is a combination of M cameras included in the N cameras; and the first evaluation value of M cameras included in the camera group; Calculating a third evaluation value of the camera group based on the second evaluation value of the camera group; and selecting the M cameras included in the camera group having the highest third evaluation value; May be included.

  Thereby, a camera can be appropriately selected based on the evaluation value based on the video imaged by each camera and the evaluation value based on the position and shooting direction of the camera.

  In addition, the video distribution system according to one aspect of the present invention provides M (N is a natural number smaller than N) as the video source of the video to be displayed from N (two or more natural numbers) cameras capturing the same scene. A video distribution system for selecting the number of cameras, the acquisition unit acquiring the positions and shooting directions of the N cameras, and the number of the N cameras based on the acquired positions and shooting directions of the N cameras. And a selection unit that selects the M cameras from the cameras.

  Thereby, the said video delivery system can select a camera appropriately from several cameras based on the position and imaging | photography direction of a camera.

  Note that these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, and computer program. Also, any combination of recording media may be realized.

  Hereinafter, embodiments will be specifically described with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present invention. The numerical values, shapes, materials, constituent elements, arrangement positions and connecting forms of the constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

(Embodiment)
(1) In the camera selection method according to the present embodiment, M images that can be displayed by a display application or the like are shot from a large number of N parameter cameras based on the estimated camera position and camera orientation (shooting direction). Select the camera you are using. In the camera selection method, it is determined whether or not camera switching is necessary for the initially selected M cameras, and if necessary, a new camera is selected from the NM cameras.

  Further, in the camera selection method, when the entire selected camera is reset, a new initial selection is performed.

  In addition, the number of parameter cameras N and the number of cameras M selected in the initial selection may be increased or decreased at any time.

  (2) The estimation of the camera position and camera orientation includes a method using N or less videos, a method using camera sensor information, or a method using both.

  (3) As an initial selection method, there are a method of photographing a photographing target space without blind spots, a method of photographing a specific subject from a plurality of directions, or a method including both.

  (4) Check the scenes in the M images or the movement of the camera at any time interval, and determine whether to switch the camera. For example, the number of cameras to be switched is 1 or more and M or less. Also, when switching one camera, a camera that is shooting a scene close to the scene shot by the camera before switching is selected. Two or more cameras may be newly selected for one camera before switching.

  When switching between more than 1 and less than M cameras, the same selection process as that for switching one camera is performed for each camera to be switched. Further, when switching the M cameras, the same processing as the initial selection is performed. At this time, the value of M may be increased or decreased.

  As a result, even when the number of viewable video parameters is enormous, distribution of multi-view video content that is highly valuable to viewers without burdening viewers, server managers, video managers, and video supervisors Can be realized.

  First, the configuration of the video distribution system 100 according to the present embodiment will be described. FIG. 1 is a block diagram showing an overall configuration of a video distribution system 100 according to the present embodiment. This video distribution system 100 includes a plurality of cameras 101, a terminal device 102, and a server 103, each of which can communicate via a network 104A or 104B.

  The plurality of cameras 101 generate a plurality of viewpoint videos by photographing the same scene from different viewpoints in the same time zone. Each camera 101 is carried by each of a plurality of users. For example, the plurality of cameras 101 are owned by a plurality of spectators in a place such as a sports stadium. In addition, each camera 101 performs video shooting and audio acquisition. In addition, each camera 101 acquires sensor information other than the image indicating the position information and posture information (shooting direction) of the camera 101 at the same time as shooting the image using a GPS, WiFi, gyro sensor, or the like. The camera 101 may be any device having at least a photographing function, and is, for example, a digital still camera, a digital video camera, a smartphone, or a mobile terminal. The plurality of cameras 101 may include not only cameras owned by the audience but also fixed cameras or broadcasting cameras. In addition, each camera 101 transmits the captured viewpoint video, audio, and sensor information to the server 103 via the network 104A.

  Moreover, in this Embodiment, the some camera 101 is a digital camera, a smart phone, etc. which are carried by the user, for example, and the case where the kind (function) of each camera 101 is different, respectively is assumed.

  The networks 104A and 104B are, for example, the Internet. In FIG. 1, the networks 104A and 104B are individually described. However, a plurality of cameras 101, the terminal device 102, and the server 103 may be connected via a single network. Further, part or all of communication between devices may be performed directly without going through the network 104A or 104B. The connection between the devices may be either a wired connection or a wireless connection.

  The server 103 temporarily holds viewpoint videos shot by the plurality of cameras 101. In addition, the server 103 distributes a part of the plurality of viewpoint videos to be held to the terminal device 102 via the network 104B in accordance with an instruction from the user 106 via the terminal device 102. At this time, the server 103 selects a number of videos that can be reproduced by the terminal device 102 from among a large number of viewpoint videos by a selection method using camera information. The camera information indicates the position, direction, and focal length of the camera. The camera information is information registered in advance in the server, sensor information acquired at the same time as the video, information calculated by processing the video, or information calculated using the sensor information and the video information. The selection method is a method of photographing the photographing target space without blind spots, a method of photographing a specific subject from a plurality of directions, or both. The camera information may include information on the angle of view (zoom magnification) in addition to the above information.

  Further, the server 103 performs processing such as switching of the viewpoint video according to an instruction from the user 106 or an analysis result of a shooting scene or camera motion.

  The terminal device 102 receives the viewpoint video from the server 103 according to the instruction of the user 106 and outputs the viewpoint video to the monitor 105 by a method according to the instruction of the user 106. The monitor 105 may be any output device such as a PC monitor, a tablet terminal, a smartphone, a mobile phone, or a notebook PC monitor. Here, the terminal device 102 and the monitor 105 are individually described, but the monitor 105 may be included in the terminal device 102.

  The instruction from the user is performed by, for example, a screen touch operation or a mouse, but any input device may be used.

  The user instruction is transmitted to the server 103. The user's instruction is used as a trigger for starting video distribution on the server 103 or switching the viewpoint video. Further, the server 103 determines a video selection method based on a user instruction.

  Note that the audio data is not necessarily acquired by all the cameras 101. The video distribution system 100 may include a microphone that acquires only audio. The server 103 may distribute the audio added to the selected video as it is to the terminal device 102 or select the audio acquired by the camera 101 (or microphone) closest to the target area or the target subject. Alternatively, a voice with good sound quality may be selected from the acquired voices, or a voice to be distributed may be created by synthesizing a plurality of voices.

  A plurality of viewpoint videos may be transmitted from the plurality of cameras 101 in real time, and the user 106 may view the viewpoint videos in real time using the terminal device 102, or at least one of video transmission and viewing may be viewed in real time. It does not have to be performed. That is, the user 106 may view a viewpoint video shot in the past at an arbitrary timing. Further, the transmission and reception of video signals (video) described below mainly means stream transmission and reception in which video signals are continuously transmitted or received.

  Next, the configuration of the server 103 will be described. FIG. 2 is a block diagram illustrating the configuration of the server 103. The server 103 includes a reception unit 111, a storage unit 112, a control unit 113, a camera information calculation unit 114, an initial camera selection unit 115, a camera switching determination unit 116, a switching camera selection unit 117, and a reset determination. Unit 118 and transmission unit 119.

  The receiving unit 111 receives viewpoint video and sensor information transmitted from the plurality of cameras 101. The accumulating unit 112 adds ID information for identifying the transmission source camera 101 to the viewpoint video and sensor information received by the receiving unit 111 and stores them. The sensor information is information output from WiFi, GPS, a gyro sensor, or the like, and is information that can specify a camera position and a camera posture. Further, the storage unit 112 stores the camera position and the camera posture calculated by the camera information calculation unit 114 using the viewpoint video and the sensor information.

  Note that the storage unit 112 may store the viewpoint video and sensor information received by the reception unit 111 as they are. For example, in order to perform a three-dimensional reconstruction process in the camera information calculation unit 114, a plurality of viewpoint videos may be stored. Each may be divided into frames, and frames at the same time may be stored as one set.

  The control unit 113 controls each processing unit.

  The camera information calculation unit 114 acquires a plurality of viewpoint videos from the storage unit 112, and calculates a camera position and a camera posture by a three-dimensional reconstruction method. The camera information calculation unit 114 may acquire sensor information from the storage unit 112 and calculate the camera position and camera posture using the sensor information. Alternatively, the camera information calculation unit 114 may calculate the camera position using both viewpoint video and sensor information. In addition, the camera posture may be calculated. The calculated camera position and camera posture are stored in the storage unit 112.

  The initial camera selection unit 115 selects a selected camera having the number of viewpoints to be provided to the user from the enormous number of cameras 101 that are photographing the subject. The initial camera selection unit 115 uses the camera position and camera posture calculated by the camera information calculation unit 114 for this selection.

  The camera switching determination unit 116 confirms whether or not the space or subject to be captured is captured by the camera capturing the image being displayed, and determines that the camera is switched if not captured.

  The switching camera selection unit 117 selects a new camera when the camera switching determination unit 116 determines to switch the camera. The switching camera selection unit 117 uses the camera position and camera posture calculated by the camera information calculation unit 114 for this selection. Moreover, the switching camera selection part 117 reduces the number of selection cameras, when an appropriate camera does not exist.

  The reset determination unit 118 performs initial camera selection again when a reset instruction is issued from the user 106, or when the number of selected cameras is 0 or less than a threshold value.

  Hereinafter, the operation of the server 103 will be described. First, the process of the camera information calculation unit 114 will be described. FIG. 3 is a flowchart of camera information calculation processing by the camera information calculation unit 114.

  First, the camera information calculation unit 114 acquires, as input processing, viewpoint videos, sensor information, or both taken by a plurality of cameras 101 from the storage unit 112 (S101). Note that which information is acquired is instructed by the control unit 113. Further, the camera information calculation unit 114 may divide a plurality of viewpoint videos into frames in order to perform a three-dimensional reconstruction process, and create an image set of frames at the same time.

  When the information acquired by the input process is a viewpoint video (Yes in S102), the camera information calculation unit 114 performs a three-dimensional reconstruction process (S103). Specifically, the camera information calculation unit 114 calculates a camera position and a camera posture by performing three-dimensional reconstruction of each camera using a plurality of viewpoint videos. In the three-dimensional reconstruction, the translation vector and rotation matrix of each camera in a three-dimensional coordinate space composed of three axes of x, y, and z are calculated using epipolar geometry as a constraint condition. A specific example of the three-dimensional reconstruction based on the epipolar geometry will be described later in detail with reference to FIG. The translation vector T and the rotation matrix R are expressed by the following (formula 1) and (formula 2), respectively. The translation vector T indicates the camera position in the three-dimensional space, and the rotation matrix R indicates the tilt of the camera with respect to each axis in the three-dimensional space, that is, the camera posture. Α, β, and γ indicate angles obtained by rotating the camera around the x, y, and z axes, respectively.

  When the information acquired by the input process is sensor information (No in S102 and Yes in S104), the camera information calculation unit 114 calculates the camera position and camera posture using sensor information such as WiFi, GPS, or a gyro sensor. (S105). For example, the camera information calculation unit 114 sets a three-dimensional coordinate space, and calculates the coordinates of the camera position in the three-dimensional coordinate space and the tilt of the camera with respect to each axis in the three-dimensional coordinate space, that is, the camera posture.

  Next, the camera information calculation unit 114 stores the camera position and camera posture calculated in step S103 or S105 in the storage unit 112 (S106).

  In addition, a series of processes in steps S101 to S106 are repeatedly executed at predetermined time intervals.

  Note that the processing order of the video input determination (S102) and the sensor information input determination (S104) is not limited to this order, and may be the reverse order, or a part or all of them may be performed in parallel. Good.

  Hereinafter, the principle of three-dimensional reconstruction will be described with reference to FIG.

  For example, in the three-dimensional reconstruction, the coordinates and orientation of the camera in the world coordinate system are calculated using epipolar geometry as a constraint condition. Furthermore, the three-dimensional position in the world coordinate system of the point on the image image | photographed with the camera is calculated.

Here, 0 shown in FIG. 4 is the origin of the world coordinate system, T ₁ , T ₂ , and T ₃ are the camera coordinates of the camera 1, camera 2, and camera 3 in the world coordinate system, and R ₁ , R ₂ and R ₃ are inclinations of the camera coordinate system of the camera 1, the camera 2 and the camera 3 with respect to the world coordinate system, M is a point on the subject, and m ₁ , m ₂ and m ₃ are the camera 1 and the camera 2 is the position of the point M on the subject in the images 1, 2, and 3 taken by the camera 3.

In order to acquire the position and shooting direction of each camera, it is necessary to calculate the rotation matrix and translation vector of the camera in the world coordinate system. First, a method for calculating the rotation matrix and the translation vector of image 1 and image 2 will be described. When the point m ₁ = (u ₁ , v ₁ , 1) on the image 1 and the point m ₂ = (u ₂ , v ₂ , 1) on the image 2 correspond to each other, the epipolar equation m ₁ The relationship ^T Fm ₂ = 0 holds. Here, F is referred to as Fundamental Matrix (F matrix).

  Each point can be acquired as (Equation 4) and (Equation 5) which are points of each camera coordinate system by (Equation 3), which is a conversion equation using the internal parameter K of each camera, and the epipolar equation becomes (Equation 6). Can be rewritten as

Here, E is called an Essential matrix (E matrix). Further, each element of the E matrix can be calculated using a plurality of corresponding points. Further, after calculating each element of the F matrix using a plurality of corresponding points such as the points m ₁ and m ₂ between the images, the E matrix may be acquired by the conversion equation EK ⁻¹ FK. By decomposing this E matrix, the rotation matrix and translation vector from image 1 to image 2 can be acquired in the world coordinate system.

  When the position of the camera 1 in the world coordinate system and the tilt of the camera 1 with respect to each axis of the world coordinate system are known, the positions and orientations of the camera 1 and the camera 2 in the world coordinate system can be acquired using the relative relationship. The position and orientation of the camera 1 in the world coordinate system may be calculated from sensor information other than video, or may be measured in advance. Note that the position and orientation of another camera may be calculated using the camera coordinate system of the camera 1 as the world coordinate system.

  In addition, a three-dimensional point M on the world coordinate system can be acquired by a triangle formed using the rotation matrix and translation vector of image 1 and image 2.

In this embodiment, the above geometric relationship is expanded to three or more viewpoints. Specifically, as an example of adding image 3 to image 1 and image 2, E matrix is also calculated for image 2 and image 3 and image 1 and image 3, respectively, and the relative rotation matrix between the cameras and By acquiring the translation vectors and integrating them, the rotation matrix and translation vector in the world coordinate system of the camera of the image 3 can be calculated. Further, the rotation matrix and the translation vector of the image 3 may be calculated from the corresponding points in each of the image 3 and the images 1 and 2. Specifically, corresponding points are obtained from image 1 and image 3 and image 2 and image 3. Here, if the m ₃ of the image 3 corresponding to the m ₂ on m ₁ and the image 2 on the image 1 is obtained, since the three-dimensional coordinates M of corresponding points are acquired, the image 3 The correspondence between the points and the coordinates in the three-dimensional space can be acquired. At this time, the following relationship (Formula 7) holds.

  Here, P is referred to as Perspective matrix (P matrix). Since the relationship P = KE is established between the P matrix, the E matrix, and the inner matrix, the E matrix of the image 3 can be acquired. Thereby, a rotation matrix and a translation vector are obtained.

  Hereinafter, video selection processing by the server 103 will be described. FIG. 5 is a flowchart of video selection processing by the server 103.

  First, the initial camera selection unit 115 performs an initial camera selection process (S111). Specifically, the initial camera selection unit 115 selects an arbitrary number of cameras to be provided to the user from the enormous number of cameras 101 that are photographing the subject. The initial camera selection unit 115 uses the camera position and camera posture calculated by the camera information calculation unit 114 for this selection.

  Next, the camera switching determination unit 116 performs a camera switching determination process (S112). Specifically, the camera switching determination unit 116 checks whether or not the space or subject to be captured is captured by the camera that is capturing the displayed video, and determines that the camera is switched if not captured.

  If it is determined in step S112 that the cameras are to be switched (Yes in S113), the switched camera selection unit 117 performs a switched camera selection process for selecting a new camera (S114). Specifically, the camera switching determination unit 116 uses the camera position and camera posture calculated by the camera information calculation unit 114 for this selection.

  Note that when there is no appropriate camera, the switching camera selection unit 117 may reduce the number of cameras to be selected. Further, when there are a plurality of appropriate cameras for one camera before switching, the switching camera selection unit 117 may increase the number of cameras to be selected.

  Next, the reset determination unit 118 performs a reset determination process (S115). Specifically, the reset determination unit 118 determines to reset all currently selected cameras when a reset instruction is issued from the user, or when the number of cameras to be selected is 0 or less than a threshold value. .

  If it is determined to reset in step S115 (Yes in S116), initial camera selection is newly performed (S111). At this time, the initial camera selection unit 115 may increase or decrease the number of newly selected cameras from the number of currently selected cameras.

  In addition, a series of processes in steps S112 to S116 are repeatedly executed at predetermined time intervals.

  Hereinafter, the initial camera selection process (S111 in FIG. 5) by the initial camera selection unit 115 will be described. FIG. 6 is a flowchart of the initial video selection process (S111).

  In the present embodiment, there are three shooting modes as a shooting mode: a spatial shooting mode, a tracking shooting mode, and a hybrid shooting mode. Note that not all three shooting modes need be used, and only one or two shooting modes may be used.

  The selection of the shooting mode is controlled by the control unit 113. Specifically, the control unit 113 selects a shooting mode based on user designation, event occurrence, tracking target automatic detection, or the number of selected cameras.

  The space shooting mode is a mode in which a camera is selected for shooting a target area that is an area in a specified real space. The tracking shooting mode is a mode for selecting a camera for tracking shooting a target subject that is an object such as a moving person or object in a specified real space. The hybrid shooting mode is a mode in which both the space shooting mode and the tracking shooting mode are combined.

  When the shooting mode is the space shooting mode (Yes in S121), the initial camera selection unit 115 is a three-dimensional region corresponding to the region on the video specified by the user or a three-dimensional region determined to be important by scene analysis. One camera that captures an image including the largest target area is selected as the main camera (S122). This camera is also called a spatial main camera.

  In addition, when the three-dimensional reconstruction is performed, the initial camera selection unit 115 uses the three-dimensional model for associating the region on the video with the three-dimensional position. Further, in the scene analysis, the initial camera selection unit 115 determines that the center of the shooting target space or the entrance to the shooting target space is important.

  Next, the initial camera selection unit 115 selects a plurality of sub-cameras so that the imaging target space is imaged with equal intervals or no blind spots with the main camera as a reference (S123). These cameras are also called spatial sub-cameras.

  On the other hand, when the shooting mode is the tracking shooting mode (Yes in S124), the initial camera selection unit 115 has the largest number of target subjects that are one subject specified by the user or one subject determined to be important by scene analysis. One camera that captures the included video is selected as the main camera (S125). This camera is also called a tracking main camera. Note that the initial camera selection unit 115 selects one main camera for one target subject when there are a plurality of target subjects.

  In addition, in the scene analysis, the initial camera selection unit 115 determines that a person who is acting abnormally or a person who is most noted in a game is important.

  Note that the initial camera selection unit 115 may be based on photographing the target subject from the front instead of using the largest number of target subjects.

  Next, the initial camera selection unit 115 selects a plurality of sub cameras so as to surround the subject at equal intervals with the main camera as a reference (S126). These cameras are also called tracking sub-cameras.

  On the other hand, when the shooting mode is the hybrid shooting mode (No in S121 and No in S124), the initial camera selection unit 115 selects one spatial main camera in the same manner as the selection of the main camera in the spatial shooting mode, One tracking main camera is selected in the same manner as the main camera selection in the target shooting mode (S127).

  Next, the initial camera selection unit 115 assigns the number of sub cameras to the space shooting mode and the target shooting mode at an arbitrary ratio. Then, the initial camera selection unit 115 selects a spatial sub camera by the same method as the spatial shooting mode, and selects a tracking sub camera by the same method as the tracking shooting mode (S128).

  Note that the order of the confirmation of the spatial photographing mode (S121) and the confirmation of the tracking photographing mode (S124) is not limited to this order, and may be in the reverse order, or a part or all of them may be performed in parallel. Good.

  Hereinafter, the camera switching determination process (S112 in FIG. 5) by the camera switching determination unit 116 will be described. FIG. 7 is a flowchart of the camera switching determination process (S112).

  The camera switching determination unit 116 performs the processes of steps S131 to S133 for each of the plurality of cameras selected in the initial camera selection process.

  First, the camera switching determination unit 116 determines whether a processing target camera (hereinafter referred to as a target camera) is a space camera (a space main camera or a space sub camera) (S131).

  When the target camera is a spatial camera (Yes in S131), the camera switching determination unit 116 determines that switching is necessary when the designated shooting area is out of frame due to the movement of the target camera (S132). Specifically, the camera switching determination unit 116 sets the amount of movement of the camera position and the amount of rotation in the shooting direction in which the movement amount of the camera position and the rotation amount in the shooting direction are set in advance according to the viewing angle of the camera. When the threshold value is exceeded, it is determined that the shooting area has been out of frame.

  On the other hand, when the target camera is a tracking camera (tracking main camera or tracking sub camera) (No in S131), the camera switching determination unit 116 performs tracking camera switching determination processing (S133). For example, the camera switching determination unit 116 performs tracking using a three-dimensional model when three-dimensional reconstruction is performed in the camera information acquisition process.

  Hereinafter, the tracking camera switching determination process (S133 in FIG. 7) will be described. FIG. 8 is a flowchart of the tracking camera switching determination process (S133).

  First, the camera switching determination unit 116 determines whether to perform tracking on a two-dimensional image or to perform tracking in a three-dimensional model (S141). Specifically, the camera switching determination unit 116 determines to perform tracking on the two-dimensional image when the three-dimensional reconstruction is not performed. In addition, when performing the 3D reconstruction, the camera switching determination unit 116 performs tracking on the 2D image or tracking in the 3D model based on the allowable load of the server, the required tracking accuracy, or the user designation. Choose what to do.

  Note that the camera switching determination unit 116 may perform only one of the process of performing tracking on the two-dimensional image and the process of performing tracking within the three-dimensional model.

  When tracking is performed on a two-dimensional image (Yes in S141), the camera switching determination unit 116 captures the tracking camera that is selecting the target subject in the image specified by the scene analysis or specified by the user. The tracking of the target subject using only the time-series images is started (S142).

  The camera switching determination unit 116 associates the same subject between the current frame and the past frame at the shooting frame rate interval (S143).

  When the tracking is successful (Yes in S144), that is, when the association of the subject is successful, the camera switching determination unit 116 performs the process from step S143 on the next frame. On the other hand, if the tracking fails (No in S144), that is, if the association of the subject fails, the camera switching determination unit 116 determines that the target camera can no longer be tracked by the selected camera, and the camera needs to be switched. (S145).

  On the other hand, when tracking is performed in the 3D model (No in S141), the camera switching determination unit 116 associates the target subject with the 3D model and starts tracking the target subject in the 3D model (S146). .

  The camera switching determination unit 116 associates the same subject with the current 3D model and the past 3D model at the shooting frame rate interval. Then, the camera switching determination unit 116 projects the associated subject onto the frame at the current time of the selected tracking camera (S147).

  When the tracking is successful, that is, when the association of the 3D model between the frames is completed and the target subject is included in the frame at the current time obtained by the projection (Yes in S148), the camera switching determination The unit 116 performs the processing after step S147 for the next frame. On the other hand, when tracking fails (No in S148), that is, when matching of the 3D model between frames fails, or when the target subject is not included in the frame at the current time obtained by projection The camera switching determination unit 116 determines that the target camera can no longer be tracked by the selected camera, and determines that camera switching is necessary (S145).

  The tracking camera switching determination process and the tracking camera switching camera selection process include three cases: a case where the subject is tracked only with a two-dimensional image, a case where the subject is tracked only with a three-dimensional model, and a case where both are used. . Details of each case will be described with reference to FIGS.

  Hereinafter, the switching camera selection process (S114 in FIG. 5) will be described. FIG. 9 is a flowchart of the switching camera selection process (S114).

  The processes of steps S151 to S157 shown in FIG. 9 are performed for each camera determined to be switched.

  When the target camera is a spatial camera (Yes in S151), the switching camera selection unit 117 determines that the camera position, the camera posture, and the focal length from the plurality of candidate cameras in the arbitrarily set range are not changed. A camera closest to the camera position, camera posture, and focal length at the time of initial video selection of the spatial camera is selected as a new spatial camera (S152).

  Specifically, the following evaluation formula (Formula 8) is used.

score = w1 * (pos (cA) −pos (cB)) ^ 2
+ W2 * (dir (cA) -dir (cB)) ^ 2
+ W3 * (for (cA) -for (cB)) ^ 2 (Equation 8)

  Here, w1, w2 and w3 are weighting factors, pos (ck) indicates the camera position of the camera k, dir (ck) indicates the camera posture of the camera k, and for (ck) is the focal length of the camera k. CA represents a spatial camera before switching, and cB represents a candidate camera.

  When using the above evaluation formula, the switching camera selection unit 117 selects one of the plurality of candidate cameras having the smallest score as a new spatial camera.

  On the other hand, when the target camera is a tracking camera (No in S151), the switching camera selection unit 117 selects a switching camera using only a two-dimensional image or selects a switching camera using a three-dimensional model. Is determined (S153). This determination method is the same as that in step S141 described above, for example.

  When the switching camera selection unit 117 selects a switching camera using only a two-dimensional image (Yes in S153), the switching camera selection unit 117 estimates the position when the subject tracked by the camera switching determination unit 116 disappears from the frame. Then, the switching camera selection unit 117 selects a camera in which the subject is present at the center or the subject is shown to be the largest among the cameras capturing the estimated position (S154).

  On the other hand, when the switching camera selection unit 117 selects a switching camera using a three-dimensional model (No in S153), in the camera switching determination process, whether the subject has been tracked on the two-dimensional image, or on the three-dimensional model. It is determined whether the subject has been tracked (S155).

  When the subject is tracked on the two-dimensional image (Yes in S155), the switching camera selection unit 117 calculates a three-dimensional model of the subject of the previous frame where the subject tracked by the camera switching determination unit 116 disappeared. To do. Then, the switching camera selection unit 117 selects a camera in which the subject is present at the center or the subject is the largest among the cameras capturing the three-dimensional model (S156).

  On the other hand, when the subject is tracked on the three-dimensional model (No in S155), the switching camera selection unit 117 has the subject at the center of the cameras capturing the subject tracked by the camera switching determination unit 116. The camera that is present or the subject is the largest is selected (S157).

  In any switching method, when there is no camera suitable for switching, the switching camera selection unit 117 may not switch and reduce the number of cameras to be selected, or a plurality of cameras suitable for switching. If present, one camera may be increased to a plurality of cameras.

  Note that the processing order of camera shooting mode confirmation (S151), switching method confirmation (S153), and tracking method confirmation (S155) is not limited to this order, and may be any order. Some or all of them may be performed in parallel.

  Hereinafter, an operation for tracking a subject using only a two-dimensional image in the tracking camera switching determination process and the switching camera selection process will be described in detail. 10 and 11 are diagrams for explaining this operation.

  FIG. 10 is a diagram illustrating frames 201A, 201B, and 201C that are captured by the tracking camera at times t, t + 1, and t + 2.

  First, the camera switching determination unit 116 tracks the subject 202 at time t + 1 by associating the frame 201B at time t + 1 and the frame 201A at time t. Note that this association does not have to be performed between consecutive frames, and may be performed between two frames straddling one or more frames, or may include three or more frames including future frames such as time t + 2. It may be performed between frames.

  Specifically, the camera switching determination unit 116 performs this association using template matching of a rectangular area surrounding the subject. The camera switching determination unit 116 may integrate the template matching results for each sub-region obtained by dividing the rectangular region, and may perform association using the integrated result, or may include a plurality of feature points on the subject or Correlation between frames may be performed by associating local feature amounts.

  In addition, the camera switching determination unit 116 may perform tracking by performing online learning in which the subject area is set as the correct answer area and the surrounding area is set as the non-correct answer area.

  Further, the camera switching determination unit 116 does not perform tracking of the subject using only one selected camera, but uses a plurality of cameras including the selected camera, and integrates the tracking results of the plurality of cameras. Then, tracking may be performed using the integration result. For example, the camera switching determination unit 116 recognizes the same subject in the video of each camera by performing specific object identification using a database in which subjects photographed in advance from a plurality of viewpoints are learned.

  At time t + 2, the subject 202 is out of the frame from the tracking camera. FIG. 11 is a diagram illustrating the state of the real space at time t + 2. The camera 101A is a tracking camera that is being selected, and images the region 203A.

  Since the subject 202 is out of frame at time t + 2, the camera switching determination unit 116 determines that the camera needs to be switched.

  In addition, the switching camera selection unit 117 calculates a region 203 </ b> A that appears in the field of view of the image from the position, orientation, and focal length of the selected tracking camera calculated by the camera information calculation unit 114. Then, the switching camera selection unit 117 estimates that the framed out subject 202 is near the right side of the region 203A.

  The switching camera selection unit 117 selects the camera 101B that is shooting centering on the subject 202 or shooting the video with the largest area of the subject among the plurality of cameras that are shooting near the right side of the region 203A. Select as a new tracking camera. In this example, the camera 101B captures an area 203B including the subject 202.

  Here, in order to continue the tracking, it is necessary to specify the subject 202 in the image captured by the camera 101B. For example, the switching camera selection unit 117 performs association between the frame at time t + 1 of the camera 101A and the frame at time t + 2 of the camera 101B. For example, a method similar to the tracking method performed between frames of the camera 101A is used for this association.

  Note that, in order to correct the difference in field of view between the two cameras 101A and 101B, the switching camera selection unit 117 further uses a frame obtained by projective transformation using camera information such as the camera position, posture, and focal length. Tracking may be performed.

  The camera switching is not limited to the case where the subject is out of frame, but is also performed when the subject exists in the shooting area of the selected tracking camera but disappears from the frame due to occlusion.

  In addition, the switching camera selection unit 117 does not necessarily switch the camera immediately after the subject disappears from the frame, and may continue to estimate the motion of the subject during the disappearance and wait for it to appear in the frame again. That is, the switching camera selection unit 117 may perform camera switching when a subject is not included in a frame for a predetermined period (a plurality of frames).

  Hereinafter, in the tracking camera switching determination process and the switching camera selection process, an operation for tracking the subject using only the three-dimensional model will be described in detail. FIG. 12 is a diagram for explaining this operation.

  First, when the subject 202 to be tracked is specified on the image (frame 201A) at time t, it is necessary to specify the subject 202 on the image in the three-dimensional model.

  The camera switching determination unit 116 calculates a perspective projection matrix between the image and the three-dimensional space using the position, posture, and focal length of the currently selected tracking camera calculated by the camera information calculation unit 114. The camera switching determination unit 116 identifies the subject 202 in the three-dimensional model by projecting the feature points on the subject 202 in the image onto the three-dimensional space using the perspective projection matrix.

  At time t + 1, the camera switching determination unit 116 tracks the subject 202 by associating the three-dimensional model at time t + 1 with the three-dimensional model at time t. The association need not be performed between three-dimensional models at successive times, and may be performed between three-dimensional models at two times straddling one or more times (frames). It may be performed between three-dimensional models at three or more times including a three-dimensional model of t + 2.

  Specifically, the camera switching determination unit 116 performs the association using matching of voxels surrounding the subject. Note that the camera switching determination unit 116 may integrate the matching results for each sub-voxel obtained by dividing the voxel, and may perform association using the integrated result, or a plurality of 3D feature points or 3D on the subject. The subject may be associated by associating the local feature amount.

  In addition, the camera switching determination unit 116 may perform tracking by performing online learning in which the subject area is set as the correct answer area and the surrounding area is set as the non-correct answer area.

  Further, the camera switching determination unit 116 projects the subject 202 in the three-dimensional model at each time onto the image plane at each time during tracking, and the subject 202 exists in the field of view of the camera using the obtained image. Make sure. Specifically, the camera switching determination unit 116 performs this projection in the same manner as the association between the image and the three-dimensional model at time t.

  At time t + 2, the subject 202 is out of frame from the tracking camera. The switching camera selection unit 117 projects a three-dimensional model of the subject 202 onto the image plane of each camera in order to select a new tracking camera. The switching camera selection unit 117 performs this projection in the same manner as the association between the image and the three-dimensional model at time t.

  Then, the switching camera selection unit 117 selects the camera 101 that has captured the image in which the projected subject 202 is the center or the subject area is the largest.

  Note that, as in the case of using the two-dimensional image described with reference to FIGS. 10 and 11, camera switching is not limited to the case where the subject is out of frame, but the subject is present in the shooting area of the selected tracking camera. Is also performed when the frame disappears due to occlusion.

  In addition, the switching camera selection unit 117 may not always switch the camera immediately after the subject disappears from the frame, but may continue to estimate the motion of the subject while disappearing and wait for it to appear again in the frame.

  Hereinafter, in the tracking camera switching determination process and the switching camera selection process, an operation for tracking a subject using both a two-dimensional image and a three-dimensional model will be described in detail. FIG. 13 is a diagram for explaining the operation in this case.

  At time t, the camera switching determination unit 116 tracks the subject 202 by the same method as in the case of using the two-dimensional image described with reference to FIG.

  Since the tracking camera loses sight of the subject 202 at time t + 2, the switching camera selection unit 117 uses the latest frame 201B at time t + 1 and the three-dimensional model at time t + 1 while the subject 202 has been successfully tracked. Select a new tracking camera.

  The camera switching determination unit 116 associates the subject 202 in the frame 201B at the time t + 1 with the subject 202 in the three-dimensional model at the time t + 1 by the same method as in the case of using the two-dimensional image described in FIG. . Further, the switching camera selection unit 117 selects a new tracking camera using the three-dimensional model by the same method as in FIG. The camera switching determination unit 116 continues to track the subject 202 by the same method as in FIGS. 10 and 11 using the time-series image captured by the new tracking camera selected here.

  Similar to FIGS. 10 to 12, camera switching is not limited to the case where the subject is out of frame, but the subject is present in the shooting area of the selected tracking camera but is lost from the frame due to occlusion. Also done.

  In addition, the switching camera selection unit 117 may not always switch the camera immediately after the subject disappears from the frame, but may continue to estimate the motion of the subject while disappearing and wait for it to appear again in the frame.

  12 and 13, the camera information calculation unit 114 only calculates the camera position, posture, and focal length, and the camera switching determination unit 116 reconstructs the three-dimensional model of the subject 202. Also good. In that case, the camera switching determination unit 116 may use all the images of the camera 101 as the parameter, or may use only the image showing the subject 202, or the currently selected tracking camera and its tracking camera Only images from nearby cameras may be used.

  In tracking the subject of the tracking camera switching determination process in FIGS. 8 and 10 to 13, the camera switching determination unit 116 is suitable for tracking from the parameter camera independently of the selected camera displayed to the user. May be selected and tracked using them.

  Hereinafter, the reset determination process (S115 in FIG. 5) will be described. FIG. 14 is a flowchart of the reset determination process (S115).

  When the shooting mode is the space shooting mode (Yes in S161), the reset determination unit 118 determines that the camera selection needs to be reset when any one of the following four conditions is satisfied (S162). (1) The user has designated switching to another shooting mode. (2) Another target area is designated by the user. (3) Another important area (target area) was specified by scene analysis. (4) The number of space cameras is 0, the lower limit number or less, or the upper limit number or more.

  When the shooting mode is the tracking shooting mode (Yes in S163), the reset determination unit 118 determines that the camera selection needs to be reset if any one of the following four conditions is satisfied (S164). (1) The user has designated switching to another shooting mode. (2) Another target subject is designated by the user. (3) Another important subject (target subject) was designated by scene analysis. (4) The number of tracking cameras is 0, below the lower limit, or above the upper limit.

  When the shooting mode is the hybrid shooting mode (No in S161 and No in S163), the reset determination unit 118 is satisfied when at least one of the same conditions as the spatial shooting mode and the same conditions as the tracking shooting mode is satisfied. It is determined that the camera selection needs to be reset (S165).

  Hereinafter, a specific example of camera selection in the space shooting mode in an event venue (for example, a sports stadium or a live venue) will be described. 15 and 16 are diagrams for explaining a specific example of camera selection in the space shooting mode at the event venue.

  First, an example of selecting a plurality of selected cameras so that the camera intervals are equal will be described with reference to FIG.

  The initial camera selection unit 115 is a camera that captures a video that includes the most target region that is a three-dimensional region corresponding to a region on the video specified by the user or a three-dimensional region determined to be important by scene analysis. 101A is selected as the main camera. In addition, the initial camera selection unit 115 selects the sub cameras so that the shooting directions are evenly spaced with respect to the main camera.

  In addition, the server 103 may output information indicating whether each selected camera is a main camera or a sub camera together with a camera selection result. This information is used by the display app.

  As shown in FIG. 15, an event venue where spectator seats are installed so as to surround the basket court 360 ° will be described as an example.

  The initial camera selection unit 115 selects the sub cameras 101B to 101E so that the seats are evenly spaced with reference to the seat position of the main camera 101A so that the user can view the watching scenes from various seats. The main camera 101A may be the camera closest to the position specified by the user, may be the camera that is photographing the center specified or the largest position specified by the user, or is the most visible at the position where the venue is most visible. It may be a close camera. The visibility of the venue is preset.

  The sub cameras 101B to 101E are selected based on the main camera 101A and the center position of the event venue.

  For example, when the initial camera selection unit 115 selects five cameras in combination with the main camera and the sub camera, assuming that the center of the center circle at the center of the event venue is the origin and the main camera is set to 0 °, The sub camera is selected so that the interval is about 72 °. The direction with respect to the origin is calculated based on the camera position and the camera direction. The origin does not necessarily have to be the center of the event venue, but may be the goal, the position of the electric bulletin board or the bench, and the like.

  Alternatively, the initial camera selection unit 115 calculates the camera interval for all combinations that select five cameras from the designated position, and the five cameras are arranged most evenly with the center of the center circle as the origin. You may select a combination. At this time, it is not necessary to distinguish each camera into main and sub.

  Next, an example of selecting a camera based on the shooting position will be described with reference to FIG. As shown in FIG. 16, an event venue where spectator seats are installed so as to surround the basket court 360 ° will be described as an example.

The initial camera selection unit 115 selects a spatial camera so that the user can view the watching scenes from various seats so that the basketball court as an event venue can be photographed without blind spots.
For example, when five cameras are selected, the initial camera selection unit 115 sets a shooting field of view that is a field of view for each of the five cameras. These photographing fields may be selected by the user, or the field of view may be set so that the initial camera selection unit 115 divides the basket court into five equal parts.

  The initial camera selection unit 115 selects a camera that captures a field of view closest to each field of view as a spatial camera. The initial camera selection unit 115 calculates the field of view of the camera from the camera position, the camera posture, and the focal length.

  Note that the initial camera selection unit 115 may set the photographing field of view so that the images are seamlessly connected when the selected camera image is switched. In addition, when the selected video is synthesized to generate a video with a wide viewing angle and the video can be displayed by the display application, the initial camera selection unit 115 performs a plurality of shootings so that the shooting fields of view overlap each other so as to be suitable for synthesis. A field of view may be set.

  A specific example of camera selection in the tracking shooting mode at the event venue will be described below. 17, 18 and 19 are diagrams for explaining specific examples of camera selection in the tracking shooting mode at the event venue.

  FIG. 17 is a diagram for explaining an initial camera selection process when the target subject is one person. As shown in FIG. 17, a case where attention is paid to one target subject in a basket game will be described.

  For example, the initial camera selection unit 115 specifies a person holding the ball by specifying a subject by the user or by scene analysis, and selects one notable player who is the specified person as the subject 202 (target subject).

  The initial camera selection unit 115 selects the camera 101A that is shooting the player of interest in front or the largest image as the main camera, and selects the sub cameras 101B to 101E so that the shooting directions are uniform with respect to the main camera. For example, when selecting five cameras, the initial camera selection unit 115 selects the sub-cameras at intervals of about 72 ° when the position of the player of interest is the origin and the main camera is oriented at 0 °. The initial camera selection unit 115 calculates an orientation with respect to the origin based on the camera position and the camera direction. The target subject is not limited to a person but may be a goal or an electric bulletin board.

  FIG. 18 is a diagram for explaining a switching camera selection process when there is only one subject in the tracking shooting mode at the event venue. As shown in FIG. 18, a case where attention is paid to one target subject in a basket game will be described.

  In the tracking shooting mode, the switching camera selection unit 117 tracks the subject, and switches the camera when the current tracking camera can no longer capture the subject. For example, when the subject is likely to disappear from the field of view of a tracking camera, the switching camera selection unit 117 searches for a camera that reflects the subject in the center of the field of view according to the tracking result of the subject, and is obtained by the search. Set the new camera as the new tracking camera instead of the original tracking camera.

  When performing three-dimensional reconstruction using multi-viewpoint images, the subject is reconstructed as a three-dimensional model, so the switching camera selection unit 117 tracks the subject in the three-dimensional space. The switching camera selection unit 117 switches the tracking camera to the camera that displays the three-dimensional point on the subject at the center of the visual field or the largest image.

  When the 3D reconstruction using the multi-view video is not performed, the switching camera selection unit 117 tracks the subject by the 2D video. In the tracking at this time, the switching camera selection unit 117 uses the past frame and the current frame to predict the next movement of the subject, and switches the tracking camera to the camera that is shooting the destination space of the subject. .

  For example, when the subject 202 moves as shown in FIG. 18, the switching camera selection unit 117 tracks the subject 202.

  When reconstructing a three-dimensional model of the subject 202, the switching camera selection unit 117 performs tracking in a three-dimensional space, and when not reconstructing, predicts the motion of the subject 202 with a two-dimensional image. Then, the switching camera selection unit 117 switches the selected camera to the camera 101F that displays the subject 202 at the center of the visual field or the largest image.

  FIG. 19 is a diagram for explaining the operation when there are a plurality of subjects at the event venue. A case where attention is paid to a plurality of subjects 202A and 202B in a basket game as shown in FIG. 19 will be described.

  The initial camera selection unit 115 selects two or more attention players as the subjects 202A and 202B by specifying the subject by the user or scene analysis, and assigns at least one camera to one person. The initial camera selection unit 115 selects, as the main camera, the camera that is capturing the respective players of interest in front or most. That is, there are as many main cameras as the number of players of interest. Next, the initial camera selection unit 115 selects sub-cameras so that the shooting directions are uniform with respect to each main camera.

  For example, when shooting two players of interest with five cameras, the initial camera selection unit 115 assigns three cameras to the player of interest A (subject 202A) and two players of the player of interest B (subject 202B). Assign a camera. Since one main camera is assigned to each attention player, the initial camera selection unit 115 sets the position of the attention player A as the origin and the main camera 101A as the 0 ° azimuth for the attention player A, and about 120 The sub-cameras 101B and 101C are selected so as to be at intervals. Similarly, the initial camera selection unit 115 selects the sub-camera 101E for the noted player B so that the position of the noted player B is the origin, the main camera 101D is set to 0 °, and the interval is approximately 180 °. .

  In addition, the initial camera selection unit 115 allocates a large number of selected cameras to a player with a high priority specified by the user, or a player who has a ball by scene analysis or is closer to the ball. Also, the initial camera selection unit 115 assigns priority to the selected cameras in descending order when the number of players of interest exceeds the number of selected cameras.

In addition, the camera switching determination unit 116 tracks the player of interest using the same method as in FIG.
The initial camera selection unit 115 may select a camera in a hybrid shooting mode that is a combination of the space shooting mode and the tracking shooting mode within the range of the number of selected cameras.

  Hereinafter, an operation example in the monitoring environment of the shopping center will be described with reference to FIGS. 20 and 21. FIG. 20 is a diagram for describing camera selection in the space shooting mode in the monitoring environment of the shopping center.

  As shown in FIG. 20, in a shopping center, a monitoring environment using a fixed camera installed on a ceiling or a pillar and a mobile camera worn by a guard will be described as an example.

  The initial camera selection unit 115 selects a camera without a blind spot so that the entire shopping center can be monitored. In addition, the initial camera selection unit 115 sets, as the main camera, a camera that is shooting a place or person designated by the user on the UI, or a place that is determined to be important in monitoring, such as an entrance of a commercial product. Then, the initial camera selection unit 115 selects a sub camera so as to compensate for a place where the main camera is not shooting.

  For example, in the example of FIG. 20, the camera 101A that captures the vicinity of the entrance is selected as the main camera, and the sub cameras 101B to 101E are selected so that the entire shopping center can be monitored.

  In addition, the initial camera selection unit 115 acquires, as prior information, an area that can be photographed by a fixed monitoring camera and an area that is a blind spot in a shopping center. In addition, you may supplement the area | region used as a blind spot, when a security guard image | photographs with a mobile camera.

  FIG. 21 is a diagram for describing camera selection in the tracking shooting mode in the monitoring environment of the shopping center.

  As shown in FIG. 21, a case will be described in which one subject 202 is focused on in monitoring a shopping center.

  The initial camera selection unit 115 identifies a subject specified by the user on the UI or a person who is abnormally acting by scene analysis, and selects at least one identified person of interest as a target subject. The initial camera selection unit 115 ranks the cameras in the shopping center based on the distance between the target person and the camera, the angle of the camera with respect to the front of the target person, or the area (number of pixels) of the target person on the image. And select as many cameras as the number of cameras selected from the top. At this time, the initial camera selection unit 115 selects the highest camera as the main camera.

  In the example illustrated in FIG. 21, the camera 101A is selected as the main camera, and the cameras 101B to 101D are selected as the sub cameras.

  Note that the target subject is not limited to a person, and an event occurrence place such as a fire place or a product fall may be set as the target subject.

  In any case of FIGS. 15 to 21, the audio distributed by the server 103 to the terminal device 102 may be audio acquired by the main camera, or audio acquired by the terminal closest to the main camera. There may be audio acquired at the terminal closest to the location specified by the user on the UI, audio acquired at the terminal closest to the subject, or audio with the best sound quality. It may be a voice obtained by synthesizing voices acquired from a plurality of terminals.

  Hereinafter, details of the initial camera selection process described in FIG. 6 and another example will be described. FIG. 22 is a flowchart of the initial camera selection process.

  First, the initial camera selection unit 115 calculates an evaluation value for a single camera (S201). Then, the initial camera selection unit 115 selects one main camera based on the calculated evaluation value. Specifically, the initial camera selection unit 115 selects the camera with the highest evaluation value as the main camera.

  Next, the initial camera selection unit 115 acquires the number of cameras to be selected (S202). For example, the initial camera selection unit 115 acquires the number of cameras designated by the user or set in advance.

  Next, the initial camera selection unit 115 calculates a combination evaluation value (camera position evaluation value) based on the camera position (S203). Specifically, the initial camera selection unit 115 calculates an evaluation value based on the camera position for each combination of the main camera and the selected number minus one sub camera.

  Next, the initial camera selection unit 115 selects a camera group using the evaluation value of the single camera and the camera position evaluation value calculated in steps S201 and S203 (S204). Specifically, the initial camera selection unit 115 selects a camera group having the highest overall evaluation value obtained from the two evaluation values.

  For example, the initial camera selection unit 115 calculates the product of the sum of the evaluation values of the individual cameras of each of the plurality of cameras included in the camera group and the camera position evaluation value of the camera group as the overall evaluation value. The method for calculating the comprehensive evaluation value is not limited to this, and any method such as weighted addition may be used.

  Finally, the initial camera selection unit 115 outputs the selected camera group (S205).

  Here, the initial camera selection unit 115 first selects the main camera, and performs determination based on the comprehensive evaluation value for the camera group including the selected camera, but the most comprehensive evaluation from all combinations. A combination having the highest value may be obtained first, and a camera having the highest evaluation value of a single camera among a plurality of cameras included in the combination may be selected as the main camera.

  Hereinafter, the evaluation value calculation process (S201 in FIG. 22) for a single camera will be described. FIG. 23 is a flowchart of an evaluation value calculation process for a single camera.

  First, the initial camera selection unit 115 acquires a point cloud (S211). Here, the point cloud includes a reconstructed three-dimensional model and camera position information.

  Next, the initial camera selection unit 115 gives a flag to the target subject (target object) or the point group of the target area on the three-dimensional model (S212).

  FIG. 24 is a diagram illustrating a selection example of the target area 211. FIG. 25 is a diagram illustrating a selection example of the target subject 212. As a method for selecting a target subject or target region, there are a method in which a user manually selects a target subject or target region, and a method in which a target subject or target region is automatically selected.

  When the user manually selects the target subject or target region, the user selects the target subject or target region from the UI. Then, the initial camera selection unit 115 selects a target subject or target region on the three-dimensional model by back projecting the selected region on the two-dimensional plane onto the three-dimensional model.

  When the target subject or target area is automatically selected and the server 103 has acquired map information in advance, the initial camera selection unit 115 selects a priority monitoring area such as an entrance / exit as the target area. Alternatively, the initial camera selection unit 115 automatically detects a suspicious person by posture recognition or the like, and selects the suspicious person as a target subject.

  When the server 103 has not acquired map information in advance, the initial camera selection unit 115 determines that an area with a higher human flow rate is more important, and selects an area with a higher human flow rate as a target area. Alternatively, the initial camera selection unit 115 automatically detects a suspicious person by posture recognition or the like, and selects the suspicious person as a target subject.

  Next, the initial camera selection unit 115 creates an image by projecting the point group onto a two-dimensional plane using the camera position information (S213).

  Next, the initial camera selection unit 115 extracts a region to which a flag is assigned in each projected image (S214).

  Next, the initial camera selection unit 115 calculates the evaluation value of the image (camera) by calculating the evaluation value of the extracted region in each image (S215).

  For example, the initial camera selection unit 115 increases the evaluation value as the size of the extracted region in the image increases. Further, the initial camera selection unit 115 may increase the evaluation value as the ratio of the visualization region increases. Here, the visualization area is a ratio of an area where the extracted area can be actually confirmed in the image, and this ratio decreases when another object or the like exists in front of the target area or the target subject. The initial camera selection unit 115 may assign a priority to each part of the target area or the target subject, and increase the evaluation value as more or more high priority parts appear. For example, the initial camera selection unit 115 may increase the evaluation value when the face of the subject is captured. Further, the initial camera selection unit 115 may increase the evaluation value as the image sharpness or the like is higher or the distortion is lower.

  Hereinafter, the camera position evaluation value calculation process (S203 in FIG. 22) will be described. FIG. 26 is a flowchart of camera position evaluation value calculation processing.

  First, the initial camera selection unit 115 acquires a selection mode (S231). Here, the selection mode includes an overhead mode and a concentration mode. For example, the initial camera selection unit 115 acquires a selection mode designated by the user or set in advance.

  As shown in FIG. 27, the bird's-eye view mode is a mode for selecting the cameras 101A to 101D that are photographing the target subject 212 (or target region) at equal intervals from all directions. As shown in FIG. 28, the concentration mode is a mode for intensively selecting the cameras 101A to 101D that are photographing the target subject 212 (or target region) from a specific direction. For example, the bird's-eye view mode is effective when spectator seats are provided in all directions of the stadium for sports watching or the like. Further, the concentrated mode is effective when a spectator seat is provided only in one direction of the stage, such as a concert, or when the user wants to watch videos from a specific viewpoint direction in a concentrated manner.

  Next, the initial camera selection unit 115 extracts a plurality of camera groups including the selected number of cameras (S232). Next, the initial camera selection unit 115 calculates the camera position evaluation value of each camera group according to the selection mode (S233).

  Specifically, in the bird's-eye view mode, the initial camera selection unit 115 determines the camera position evaluation value according to whether or not a plurality of cameras are present at a certain angle from the target representative point. Specifically, the initial camera selection unit 115 increases the evaluation value as the plurality of cameras are equally spaced. Further, the initial camera selection unit 115 may prioritize the position information when the area of the area where the target subject is captured with respect to the imaging range of the camera is higher than a predetermined value. That is, the initial camera selection unit 115 may lower the evaluation value when the target subject is too high.

  In the case of the concentrated mode, the initial camera selection unit 115 determines the camera position evaluation value according to how many cameras exist in the designated direction. Specifically, the initial camera selection unit 115 increases the evaluation value as a plurality of cameras are concentrated in the designated direction.

  Note that the initial camera selection unit 115 may increase the evaluation value as the subject is facing the camera in consideration of the orientation of the subject (such as the direction of the face). In this case, for example, the orientation of the subject can be detected by face recognition or the like.

  Further, the following methods can be used as the method of extracting the camera group (S232) and calculating the camera position evaluation value (S233).

  For example, the initial camera selection unit 115 extracts all combinations of cameras as a camera group. Alternatively, the initial camera selection unit 115 classifies a plurality of cameras so that camera groups with similar configurations (such as close positions) are in the same class, and all combinations of representative cameras of each class are used as camera groups. It may be extracted. Alternatively, the initial camera selection unit 115 may select a main camera based on an evaluation value of a single camera, and extract all camera combinations including the main camera as a camera group.

  Further, the following method may be used as the camera switching determination in the time direction. For example, the camera switching determination unit 116 holds information on a previously selected camera group. The camera switching determination unit 116 holds the camera group evaluation value at each time and selects whether to switch. At this time, a parameter for controlling the switching frequency is prepared in the time direction, and the camera switching determination unit 116 determines whether to perform switching using the parameter. Specifically, the camera switching determination unit 116 performs control so that the switching is not performed as the time elapsed since the previous switching is shorter by using the parameter. For example, the camera switching determination unit 116 determines that the switching is not performed when the elapsed time from the previous switching is less than the first time, and when the elapsed time from the previous switching is less than the first time and less than the second time, the first reference In the case of the second time or more, the determination may be made based on the second reference that is looser than the first reference (easily determined by switching from the first reference).

  In the description of FIG. 5 described above, the server 103 performs camera switching and resetting when the camera switching or resetting is necessary after the initial camera selection process (S111). Only the selection process may be repeated. Also in this case, the same parameters as described above may be used so that switching is not performed frequently.

  As described above, the camera selection method according to the present embodiment uses M (N is a natural number greater than or equal to 2) cameras 101 that are shooting the same scene, and M (M is N from the source of the video to be displayed) 29 is a camera selection method for selecting (small natural number) cameras 101 (selected cameras), and performs the processing shown in FIG.

  First, the server 103 acquires the positions and shooting directions of the N cameras 101 (S241). Specifically, the server 103 calculates the position and shooting direction of the N cameras 101 by performing three-dimensional reconstruction using images captured by the N cameras 101. Alternatively, the server 103 uses the information obtained from a sensor (for example, GPS, WiFi, or gyro sensor) that is transmitted from each of the N cameras 101 and includes the position of the N cameras and The shooting direction is calculated.

  Next, the server 103 selects M cameras 101 from the N cameras 101 based on the acquired positions and shooting directions of the N cameras 101 (S242). Specifically, the server 103 selects the M cameras 101 based on the positions and shooting directions of the N cameras 101 so that the M cameras 101 can shoot the target space without blind spots. That is, based on the positions and shooting directions of the N cameras 101, the server 103 has less overlap of shooting spaces of the M cameras 101 and the shooting spaces of the M cameras 101 cover the target space. The M cameras 101 having a larger value are selected with priority. Alternatively, the server 103 preferentially selects the M cameras 101 that are shooting the target space or the target subject from a plurality of directions at equal intervals based on the positions and shooting directions of the N cameras 101.

  In step S241, the server 103 further acquires the focal lengths of the N cameras 101. In step S242, the server 103 acquires N focal lengths based on the acquired positions, shooting directions, and focal lengths of the N cameras 101. M cameras 101 may be selected from the cameras 101.

  Thereby, the camera selection method can automatically select an appropriate camera from a plurality of cameras based on the position and shooting direction of the camera.

  Further, as illustrated in FIG. 5, the server 103 further determines whether or not to switch a part of the selected M cameras 101 to another camera in the frame after step S242 (S111). Is determined (S112). Specifically, as illustrated in FIG. 8, the server 103 tracks the target subject between frames, and determines that the camera to be selected is switched when tracking of the target subject fails. More specifically, the server 103 associates the three-dimensional model of the target subject between the frames, projects the correlated three-dimensional model of the target subject onto the frame at the current time, and the target subject is included in the obtained frame. If it does not exist, it is determined that the tracking has failed (S147).

  In addition, when it is determined in step S112 to switch (Yes in S112), the server 103 selects a new camera instead of the part of the cameras based on the positions and shooting directions of the N cameras 101. (S114).

  Further, the server 103 further determines whether or not to perform step S242 (S111) again in the frame after step S242 (S111) (S115). When it is determined in step S115 that step S242 (S111) is performed (Yes in S116), the server 103 determines from the N cameras 101 to the M cameras 101 based on the positions and shooting directions of the N cameras 101. Is selected again (S111).

  In step S112, if the elapsed time from the previous switching is less than the first time, it is determined that the switching is not performed, and the elapsed time is less than the second time that is longer than the first time and longer than the first time. In such a case, whether or not to switch the some cameras to another camera is determined based on the first reference. If the elapsed time is equal to or longer than the second time, whether or not to switch the some cameras to another camera. May be determined based on a second criterion that is looser than the first criterion.

  Also, as shown in FIG. 22, the server 103, in step S242 (S111), based on the video captured by the N cameras, the first evaluation value of each of the N cameras (evaluation value of a single camera). Is calculated (S201). In addition, the server 103 determines the second evaluation value (camera position evaluation value) of each camera group that is a combination of the M cameras 101 included in the N cameras 101 based on the positions and shooting directions of the N cameras 101. ) Is calculated (S203). Then, the server 103 calculates a third evaluation value of the camera group based on the first evaluation value of the M cameras included in the camera group and the second evaluation value of the camera group. M cameras included in the camera group with the highest value are selected (S204).

  Although the camera selection method and the video distribution system according to the embodiment have been described above, the present invention is not limited to this embodiment.

  Each processing unit included in each device included in the video distribution system according to the above embodiment is typically realized as an LSI that is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.

  Further, the circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  In each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

  In other words, each device included in the video distribution system includes a processing circuit and a storage device (storage) electrically connected to the processing circuit (accessible from the processing circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuit includes a program execution unit, the storage device stores a software program executed by the program execution unit. The processing circuit executes the camera selection method according to the above embodiment using the storage device.

  Furthermore, the present invention may be the software program or a non-transitory computer-readable recording medium on which the program is recorded. Needless to say, the program can be distributed via a transmission medium such as the Internet.

  Moreover, all the numbers used above are illustrated for specifically explaining the present invention, and the present invention is not limited to the illustrated numbers.

  Further, the order in which the steps included in the above-described camera selection method and the like are executed is for illustration in order to specifically describe the present invention, and may be in an order other than the above. Also, some of the above steps may be executed simultaneously (in parallel) with other steps.

  As described above, the video distribution system and the camera selection method according to one or more aspects of the present invention have been described based on the embodiment, but the present invention is not limited to this embodiment. Unless it deviates from the gist of the present invention, the embodiment in which various modifications conceived by those skilled in the art have been made in the present embodiment, and forms constructed by combining components in different embodiments are also applicable to one or more of the present invention. It may be included within the scope of the embodiments.

  The present invention can be applied to a video distribution system that distributes videos taken by a plurality of cameras.

DESCRIPTION OF SYMBOLS 100 Video delivery system 101, 101A, 101B, 101C, 101D, 101E, 101F Camera 102 Terminal device 103 Server 104A, 104B Network 105 Monitor 106 User 111 Reception part 112 Storage part 113 Control part 114 Camera information calculation part 115 Initial camera selection part 115 116 Camera switching determination unit 117 Switching camera selection unit 118 Reset determination unit 119 Transmission unit 201A, 201B, 201C Frame 202, 202A, 202B Subject 203A, 203B Region 211 Target region 212 Target subject

Claims (13)

A camera selection method of selecting M (M is a natural number smaller than N) cameras from which a video to be displayed is captured from N (2 or more natural numbers) cameras capturing the same scene,
An acquisition step in which a circuit acquires the positions and shooting directions of the N cameras;
It said circuit based on the obtained position and the shooting direction of the N cameras, viewing contains a selection step of selecting the M cameras from the N cameras,
In the selection step,
Condition i) When the N cameras are viewed along the vertical direction, the M cameras surround a predetermined position, and
Condition ii) Each of the M cameras has a shooting direction for shooting the predetermined position, and
Condition iii) When the N cameras are viewed along the vertical direction, an angle formed by the shooting directions of two adjacent cameras included in the M cameras is included in the M cameras. And the angle formed by the shooting direction of the other two adjacent cameras is equal to the size.
A camera selection method for selecting the M cameras satisfying the above condition . Wherein in the obtaining step, using said images taken by N cameras by performing three-dimensional reconstruction, a camera selection method of claim 1, wherein calculating the position and the shooting direction of the N cameras. 2. The camera according to claim 1 , wherein in the obtaining step, the position and shooting direction of the N cameras are calculated using information obtained from a sensor included in the camera and transmitted from each of the N cameras. Selection method. In the obtaining step, the focal lengths of the N cameras are further obtained,
Wherein in the selection step, the obtained position of the N cameras, based on the photographing direction and the focal length, wherein from the N cameras in any one of claims 1 to 3 for selecting the M cameras Camera selection method. The camera selection method further includes:
A first determination step in which the circuit determines whether to switch a part of the selected M cameras to another camera in a frame after the selection step;
A switching step of selecting a new camera instead of the part of the cameras based on the positions and shooting directions of the N cameras when the circuit is determined to switch in the first determination step. The camera selection method according to any one of claims 1 to 4 . The camera selection method according to claim 5, wherein in the first determination step, a target subject is tracked between frames, and when the tracking of the target subject fails, it is determined that the camera to be selected is switched. In the first determination step, the three-dimensional model of the target subject is associated between frames, the associated three-dimensional model of the target subject is projected onto the frame at the current time, and the target subject exists in the obtained frame The camera selection method according to claim 6 , wherein it is determined that the tracking has failed if not. The camera selection method further includes:
A second determination step for determining whether or not the circuit performs the selection step again in a frame after the selection step;
If it is determined in the second determination step that the circuit performs the selection step, the M cameras are selected again from the N cameras based on the positions and shooting directions of the N cameras. The camera selection method according to claim 5 , further comprising a reselection step. In the first determination step, when the elapsed time from the previous switching is less than the first time, it is determined that the switching is not performed, and the elapsed time is equal to or longer than the first time and longer than the first time. When the time is less than the time, the first reference determines whether to switch the part of the camera to the other camera. When the elapsed time is the second time or more, the part of the camera is changed to the other The camera according to any one of claims 5 to 8 , wherein whether or not to switch to another camera is determined based on a second reference that is more easily determined when the part of the cameras is switched to the other camera than the first reference. Selection method. The selection step includes
The circuit calculates a first evaluation value of each of the N cameras based on an image captured by the N cameras;
The circuit calculates a second evaluation value of each camera group that is a combination of M cameras included in the N cameras based on the positions and shooting directions of the N cameras;
The circuit calculates a third evaluation value of the camera group based on the first evaluation value of the M cameras included in the camera group and the second evaluation value of the camera group;
It said circuit, camera selection method according to any one of claims 1 to 9, including the step of selecting the M cameras that the third evaluation value is included in the highest group of cameras. The N cameras are arranged so as to surround the predetermined position at 360 °.
  The camera selection method according to claim 1. The predetermined position is the center position of the event venue.
  The camera selection method according to claim 1. A video distribution system that selects M (M is a natural number smaller than N) cameras of a video to be displayed from N (natural numbers greater than or equal to 2) cameras that shoot the same scene,
An acquisition unit for acquiring the positions and shooting directions of the N cameras;
A selection unit that selects the M cameras from the N cameras based on the acquired positions and shooting directions of the N cameras ;
A transmission unit that distributes video captured by the M cameras ,
The selection unit includes:
Condition i) When the N cameras are viewed along the vertical direction, the M cameras surround a predetermined position, and
Condition ii) Each of the M cameras has a shooting direction for shooting the predetermined position, and
Condition iii) When the N cameras are viewed along the vertical direction, an angle formed by the shooting directions of two adjacent cameras included in the M cameras is included in the M cameras. And the angle formed by the shooting direction of the other two adjacent cameras is equal to the size.
A video distribution system for selecting the M cameras satisfying

Images