JP2015212876A - Video reproduction system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a reproduction system of virtual viewpoint video where a stable system runs for a long period of time.SOLUTION: A video reproduction system reproduces virtual viewpoint video obtained from image data imaged by a plurality of imaging means. When occurrence of failure of the imaging means is detected, the imaging means where the failure occurs is restarted after starting reproduction of the virtual viewpoint video obtained from the image data imaged by the other imaging means except the imaging means.

Description

  The present invention relates to a video playback system that generates and plays back a virtual viewpoint video using videos from a plurality of cameras. More specifically, the present invention relates to restart when a failure occurs in the video reproduction system.

  There has been proposed a video playback system that captures the same scene with a plurality of cameras and expresses a video viewed from a moving virtual viewpoint (virtual viewpoint video). Such a video reproduction system requires a large number of cameras and control PCs. As a method for stably operating such a system using a large number of devices for a long period of time, there is a method of replacing the processing of a device in which a failure has occurred with another device as described in Patent Document 1.

JP 2003-087834 A

  In a system that generates and reproduces a virtual viewpoint video, each camera captures a meaningful video in accordance with a scenario, so that replacement with another camera may be difficult. In such a system, simply restarting the failed device or removing the failed device will lead to degradation of the generated virtual viewpoint video or to stop playback of the video itself. There is a fear.

A video playback system according to the present invention is a video playback system that plays back a virtual viewpoint video generated from image data captured by a plurality of imaging means,
The detection means for detecting the occurrence of a fault in the imaging means and the imaging means in which the fault has occurred are re-established after the start of reproduction of a virtual viewpoint video obtained from image data taken by other imaging means other than the imaging means. And a control means for starting.

  According to the present invention, it is possible to stably reproduce a virtual viewpoint video for a long period of time by detecting a failed device and restarting the device according to the state of a screening scenario.

It is explanatory drawing of the video reproduction | regeneration system by Example 1. FIG. It is an example of the scenario data in Example 1, and screening control information. 1 is a diagram illustrating a system configuration of Embodiment 1. FIG. FIG. 6 is a diagram illustrating processing performed by a capture PC according to the first exemplary embodiment. 6 is a flowchart illustrating a process flow of a capture PC according to the first exemplary embodiment. FIG. 6 is a diagram illustrating processing performed by the image processing PC according to the first exemplary embodiment. 3 is a flowchart illustrating a flow of processing in the image processing PC according to the first exemplary embodiment. FIG. 6 is a diagram illustrating processing performed by the image display PC according to the first embodiment. 4 is a flowchart showing a flow of normal screening processing in Embodiment 1. 3 is a flowchart illustrating a flow of camera and PC validity determination processing according to the first exemplary embodiment. It is a figure which shows the process in the power supply controller in Example 1. FIG. 3 is a flowchart illustrating a process flow in a power supply controller according to the first exemplary embodiment. 3 is a flowchart illustrating a flow of virtual viewpoint video generation processing in Embodiment 1. It is a figure explaining the parameter used by the virtual viewpoint image | video production | generation process in Example 1. FIG. It is a figure explaining the coverage in Example 1. FIG. 3 is a flowchart illustrating a flow of camera selection processing according to the first exemplary embodiment. 10 is a flowchart illustrating a flow of processing in a power supply controller according to the second embodiment. It is a figure showing the system configuration | structure in another Example.

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. In addition, the structure shown in the following Examples is only an example, and this invention is not limited to the structure shown in figure.

  FIG. 1 is a diagram showing a video playback system that generates a virtual viewpoint video using a plurality of fixed imaging cameras and shows the video in real time in the present embodiment. As shown in FIG. 1A, the fixed cameras in the present embodiment are arranged so that the imaging areas of the cameras overlap along the virtual viewpoint movement route. In FIG. 1A, a camera 101 is fixed on a floor 102 in a suspended state by a metal frame (not shown). The virtual viewpoint 105 is mapped on a coordinate system including the floor 102 on the xy plane and having the height direction as the z axis. FIG. 1 (b) is a projection of FIG. 1 (a) onto a two-dimensional plane and assumes a case where there are a plurality of people. FIG. 1B shows a person 104 who is a subject.

  FIG. 2 is a diagram illustrating an example of scenario data in the present embodiment. Scenario data is data that defines which virtual viewpoint image data is used in which time-axis frame when a virtual viewpoint video is generated and reproduced. In this embodiment, for example, a short-time scenario of about 5 minutes is used as one scenario. FIG. 2A shows an example of scenario data representing one scenario. Scenario data includes virtual viewpoint coordinates for expressing the position of the virtual viewpoint, virtual viewpoint direction for expressing the direction of the virtual viewpoint, and virtual viewpoint for expressing the attitude of the virtual viewpoint for each frame during the screening time. Describes the upward vector. The virtual viewpoint coordinates and the virtual viewpoint direction are the position and orientation of the virtual viewpoint in this coordinate system. The virtual viewpoint upward vector is a parameter for designating how much the field of view is inclined with respect to the direction of the virtual viewpoint, and the virtual viewpoint direction and the virtual viewpoint upward vector are orthogonal to each other. By adding an angle of view to this, it is possible to express the field of view from a virtual viewpoint. In this embodiment, by adding lens effect parameters (distortion strength α, peripheral light amount drop strength β, details will be described later), a natural video image as if it was actually captured while moving the camera is generated.

  The scenario data holds the height of the reference plane, which is an image processing parameter used in the virtual viewpoint video generation process. The height of the reference plane affects the smoothness when the camera used is switched. For example, in the present embodiment, it is desirable to take the reference plane on the floor in the almost unmanned state. However, when there are a large number of people in the imaging area as shown in FIG. If the reference plane is taken, the video will be smoother. By holding the height of the reference plane in the scenario data, it is possible to correct the height of the reference plane according to the state of the subject. The correction of the height of the reference plane may be automatically generated from captured image data obtained by each camera, or may be manually designated.

  FIG. 3 is a diagram illustrating a configuration of a system for realizing the present embodiment. This system includes a plurality of cameras 101, a management server PC 301, a plurality of capture PCs 302, a plurality of image processing PCs 303, an image display PC 304, a HUB 305, and a power supply controller 306.

  The management server PC 301 is a PC that manages the entire system, and a console screen and operation devices such as a keyboard and a mouse are connected to operate the system.

  The same number of capture PCs 302 as the cameras 101 are prepared. The capture PC 302 is connected to each camera 101 by an HD-SDI (High Definition Serial Digital Interface) cable. Video captured by the camera 101 is transmitted to the image processing PC 303 via the capture PC 302. Here, when the output resolution of the camera 101 is full HD resolution (1920 × 1080 pixels, 30 frames per second), the amount of data becomes very large. Therefore, if all the images of the camera 101 are transmitted all at once, there is a problem that the bandwidth of the communication path is used up. Therefore, screening control information for reproducing a predetermined scenario using a camera or an image processing PC that can be used at present is generated, and a video is transmitted in accordance with this.

  FIGS. 2B and 2C are diagrams showing an example of the screening control information. The screening control information includes camera control information and PC control information. FIG. 2B shows an example of camera control information, in which a camera ID is associated with each frame. FIG. 2C shows an example of PC control information, in which an image processing PCID is associated with each frame. The screening control information describes which camera video is processed by which image processing PC for each frame from the screening start time to the end time. For example, in the present embodiment, of the screening control information, the value of the camera control information (FIG. 2 (b)) is 0, the transmission of the video data of the corresponding camera is unnecessary, 1 is the transmission of the video data of the corresponding camera, 2 means transmission of only the header information of the corresponding camera. Further, the value of the PC control information (FIG. 2C) means that 1 uses the corresponding image processing PC, and 0 means not use it. As shown in FIG. 2, the time included in the screening control information is described as a relative time with the screening start time set to 0: 00: 00.00f.

  The image processing PC 303 receives and processes the video transmitted from the capture PC 302 and transmits it to the image display PC 304.

  The image display PC 304 receives and buffers the video transmitted from the image processing PC 303 and plays it at a timing designated by the management server PC 301. Each capture PC, each image processing PC, the management server PC, and the image display PC are connected to each other via a network HUB 305, and transmission / reception of data is performed using TCP / IP.

  The power controller 306 is individually connected to the power sources of the management server PC 301, the camera 101, and the capture PC 302. The power supply controller 306 can selectively restart the camera 101 or the capture PC 302 in which some kind of failure has occurred, such as when a capture error is detected.

  Details of the functions of each PC will be described later. Since all the cameras 101 need to be synchronized, a synchronization signal (Genlock, GenLock) is supplied.

<Capture PC>
The capture PC 302 captures the video from the camera 101 and transmits it to the image processing PC 303.

  FIG. 4 is a diagram illustrating an example of the configuration of the capture PC 302. The operation of the capture PC will be described with reference to FIG. The capture PC 302 includes a camera control unit 411 that controls the camera according to the camera control information described in the screening control information 408 received from the management server PC 301. Note that the virtual viewpoint video generation processing applied by the image processing PC 303 requires lens characteristic data at the time of imaging. However, since it is known that the lens characteristic data changes depending on the setting (for example, f value, angle of view, imaging distance) of the camera, the capture PC 302 has a lens correction data acquisition processing unit that acquires lens correction data. 402. In this embodiment, by providing the lens correction value database 403, lens correction data corresponding to the setting at the time of imaging is obtained. The acquired lens correction data is added to the video data as header information and transmitted. Although various correction data such as distortion, peripheral light amount, and lateral chromatic aberration can be considered as lens correction data, in this embodiment, correction data for distortion and peripheral light amount are stored for simplicity. In addition, in order to generate a virtual viewpoint video, camera position / posture information 404 such as the position, direction, posture, and angle of view of the camera 101 is required. Therefore, camera position / posture information is also added to the image data as header information. In addition, the header information includes capture time, image size, image data type, etc. in addition to lens correction data and camera position / posture information.

  Transmission processing for sending video data from the capture PC 302 to the image processing PC 303 is performed by TCP / IP as described above. Since it is not efficient to reconnect to the image processing PC 303 every time data is transmitted, the transmission processing units 405 are prepared for the number of image processing PCs 303 to cope with them. In this embodiment, a transmission thread is used as the transmission processing unit. That is, transmission threads are prepared for the number of image processing PCs. In addition, since there is a case where the received data cannot be transmitted without delay due to a bandwidth problem of the transmission path, the transmission buffer 401 is provided for buffering for a predetermined number of seconds.

  In addition to the transmission processing unit 405, the capture PC manages the transmission data generation unit 406 that receives video from the camera and adds header and lens correction data to create transmission data, and the transmission processing unit 405. There is a management unit 407.

  FIG. 5 is a flowchart showing processing in the capture PC 302. In step S <b> 501, the transmission data generation unit 406 first confirms which transmission processing unit 405 is used with respect to the management unit 407.

  In step S502, the management unit 407 selects a destination image processing PC 303 according to the PC control information included in the screening control information 408 received from the management server PC 301 described later. Then, the transmission processing unit 405 corresponding to the selected image processing PC 303 is notified to the transmission data generation unit 406.

  In step S503, the transmission data generation unit 406 first acquires the transmission data area in the transmission buffer 401 of the transmission processing unit 405 instructed by the management unit 407.

  Next, in step S504, the transmission data generation unit 406 generates transmission data including the captured video in the acquired transmission data area.

  In step S505, the transmission data generation unit 406 outputs the generated transmission data to the transmission buffer 401, and notifies the management unit 407 of completion of data output.

  In step S506, the management unit 407 adds the data that has received the data output completion notification to the transmission queue 409, and executes transmission processing. In the transmission process, the management unit 407 determines whether the data stored in the transmission queue included in the management unit 407 is valid. This determination is made based on whether the difference between the capture time T0 of the target data and the screening start time T1 exceeds a predetermined time TL. If it is not valid, the transmission process is skipped. If valid, the management unit 407 issues a transmission instruction to the transmission processing unit 405 and executes transmission processing.

  Whether the image data included in the transmission data is to be compressed can be specified by the setting file. Here, as the image data type included in the header information, 1 is specified when compression is performed, and 0 is specified when it is not compressed. Also, 2 is specified when only the header is transmitted. It is necessary to compress video data in units of frames. For the compression, a JPEG (Joint Photographic Experts Group) format or a DXTC (DirectX Texture Compression) format may be used. DXTC can compress image data at a very high speed by using a GPU.

  Note that the transmission data generation unit 406 counts the rate at which a capture error has occurred when capturing video from the camera so that the status can be returned as capture error information 410 when a valid determination response is sent from the management server PC described later. The capture error may occur due to the fact that the distance between the camera and the capture PC is long (for example, more than 100 m) and the video cannot be acquired correctly at 30 frames per second. Alternatively, the capture error may be caused by a malfunction on the camera side or capture PC side. In this embodiment, when the number of frames that could not be acquired due to a capture error exceeds, for example, 10% of the number of frames to be captured, it is determined that “capture errors are many”.

<Image processing PC>
The image processing PC 303 processes the video received from the capture PC 302 and transmits it to the image display PC 304.

  FIG. 6 is a diagram illustrating an example of the configuration of the image processing PC 303. The operation of the image processing PC 303 will be described with reference to FIG. Similarly to the capture PCs 302, the reception processing units 601 corresponding to the number of capture PCs 302 are activated in order to maintain the connection state with all the capture PCs 302. The reception processing unit 601 can be activated as a thread corresponding to the transmission processing unit of the capture PC 302.

  FIG. 7 is a flowchart showing the processing of the image processing PC 303. First, in step S701, the reception processing unit 601 determines whether the received data is data that needs to be processed according to the image data type. If the data requires processing, the process advances to step S702, and the reception processing unit 601 applies image processing corresponding to the image data type to the received data, and stores the received data in the reception buffer 602 in step S703. In the present embodiment, when the image data type is 1, as processing according to the image data type, decompression processing of compressed data is applied to convert it into uncompressed data (image data type = 0).

  In step S704, the reception processing unit 601 stores the received data in the reception buffer 602, and then notifies the management unit 603 of the completion of data reception.

  In step S <b> 705, the management unit 603 selects frame image data for virtual viewpoint video generation according to the screening control information 604. In step S706, the management unit 603 issues a video generation instruction to the virtual viewpoint video generation processing unit 605. However, if a data output completion notification has not been received from the virtual viewpoint video generation processing unit 605 after the video generation instruction, the video generation instruction is blocked until an output completion notification is received. This is because if a plurality of virtual viewpoint video generation processes are executed at the same time, the throughput of the video generation process already being executed is reduced, and there is a high possibility that real-time playback cannot be performed. Therefore, the management unit 603 confirms and acquires the data output completion notification in the processing queue 609, and continues processing after acquiring the notification.

  In step S707, the virtual viewpoint video generation processing unit 605 that has received the video generation instruction generates virtual viewpoint video data according to the scenario data 606 using the video selected by the management unit 603 and the presentation time. Details of the processing will be described later.

  In step S708, the virtual viewpoint video generation processing unit 605 outputs the generated virtual viewpoint video data to the transmission buffer 607. In step S709, the virtual viewpoint video generation processing unit 605 notifies the management unit 603 and the transmission processing unit 608 of the completion of data output.

  In step S710, the transmission processing unit 608 receives the data output completion notification and transmits the generated virtual viewpoint video data to the image display PC 304. Capture time and image resolution are added as headers to the virtual viewpoint video data to be transmitted.

<Image display PC>
The image display PC 304 has the following functions.
(A) Reproduction of data received from image processing PC (normal screening)
(B) Backup video screening (backup screening)
The image display PC 304 normally reproduces data received from the image processing PC 303. However, there may be a case where screening cannot be performed normally due to a failure of a plurality of cameras or PCs. In such a case, switching to backup screening is performed by determination processing on the management server PC 301 described later. In the following, details of processing during normal screening and backup screening of the image display PC 304 will be described.

  FIG. 8 is a diagram illustrating an example of the configuration of the image display PC 304. The operation of the image display PC 304 will be described with reference to FIG. Similar to the image processing PC 303, the image display PC 304 has reception processing units 801 corresponding to the number of the image processing PCs 303, and buffers received data in the reception buffer 802. This PC is equipped with a large-capacity memory so that all videos to be screened can be buffered. This is necessary to generate a backup video. The reception processing unit 801 can be configured as a thread.

  FIG. 9 is a flowchart showing normal screening processing in the image display PC 304. In step S <b> 901, the reception processing unit 801 receives video data generated by the image processing PC 303 and stores it in the reception buffer 802. In step S902, the reception processing unit 801 transmits a data reception completion notification to the management unit 803 every time reception of one frame of video is completed.

  In step S903, the management unit 803 updates the received video management information in the received video management area 804 upon receiving the data reception completion notification. Here, the received video management information includes the capture time of the received video data and the identification ID of the reception processing unit 801. In step S904, the management unit 803 stands by until the screening start time, and performs a screening management process after step S905 from the screening start time to the screening end time.

  In step S905, the management unit 803 selects video data to be reproduced from the reception buffer 802. Selection of video data is performed using the capture time of each video data in the received video management area and the identification ID of the reception processing unit 801. Here, since the screening control information 806 is described in a relative time based on the screening start time, the capture time to be reproduced is calculated using the capture start time, and frame image data corresponding to this time is obtained. . In step S906, the management unit 803 notifies the image display unit 805 of the video data selected in this way.

  In step S907, the image display unit 805 displays the designated video data. Thereafter, when the reproduction process is applied to all frames and it is determined in step S908 that all frame images have been displayed, the management unit executes an end process in step S909. If there is no corresponding video data due to processing delay or the like, the display is not updated and the video displayed immediately before is displayed as it is. By such processing, it is possible to avoid that the screening time is extremely shortened due to the lack of frames or that the displayed video becomes unnatural.

  In the present embodiment, the end process of step S909 is a process of storing all the frame data in a file. This file is used as a backup video. In the case of multiple screenings, all the displayed videos may be stored, or only when there are no missing frames.

  Next, processing at the time of backup screening will be described. In the present embodiment, the latest one of the files saved as a result of the normal screening is used as the video to be backed up. Needless to say, a file designated in advance may be screened. The processing at the time of backup screening is basically the same as the processing shown in FIG. 9, and only different parts will be described. In the backup screening, the backup video data 809 is read into the memory via the backup data read processing unit 808 instead of performing the reception process in step 901. After that, since the screening process is executed in the same manner as the normal screening, detailed description is omitted. In the case of a backup screening, the file output of the screening data, that is, the processing corresponding to the end processing in step S909 is not performed.

<Management server PC>
The management server PC manages the entire system. Specifically, it has the following functions.
(A) Validity determination of PC or camera constituting system (b) Change of video buffering time in each PC (c) Generation of screening control information based on above a and b (d) Scenario data and screening control information (E) Distribution of setting files of capture PC and image processing PC Here, the validity determination process will be described with reference to the flowchart of FIG. In step S1001, the management server PC first makes an inquiry to the image display PC 304. In step S1002, the management server PC confirms whether a response is returned from the image display PC within a predetermined time TA. Here, when the image display PC 304 is operating normally, 0 is returned in response to the inquiry. If the image display PC 304 cannot return a response, the screening cannot be performed. Therefore, in step S1003, the management server PC displays an alert, prompts replacement of the equipment, and ends the process. The alert display in step S1003 can indicate that the screening cannot be executed because a failure has occurred in the image display PC, for example.

  If there is a response from the image display PC 304 in step S1002, that is, if there is no problem with the image display PC 304, the process proceeds to step S1004. In step S1004, the management server PC continues to make an inquiry to each image processing PC 303 in the same manner, and confirms whether or not it is operating normally. If the image processing PC 303 is operating normally, 0 is returned in response to the inquiry. In step S1005, if there is even one image processing PC 303 that does not respond, the management server PC proceeds to step S1011 and displays an alert. In step S1012, the management server PC instructs the image display PC 304 to perform backup screening and ends the process. Examples of alert display here include notifying from which image processing PC there is no response or notifying switching to backup screening.

  If there is a response from the image processing PC in step S1005, that is, if there is no problem with the image processing PC 303, the process proceeds to step S1006 and an inquiry is made to the capture PC 302. The capture PC 302 returns, for example, 1 when the camera cannot be recognized, 2 when the camera can be recognized but many capture errors occur, and 0 otherwise. In the present embodiment, the above case is taken as an example for the sake of simplification of explanation, but it goes without saying that a value for specifying the type of a fine failure factor may be returned. The inquiry result to the capture PC 302 can be detected by the capture error information 410 shown in FIG. That is, the capture error information is information indicating a detection result.

  In step S1007, the management server PC generates a valid camera list from the camera and capture PC states obtained as a result of the inquiry. The effective camera list is a list including a pair in which the state of the camera and the capture PC is normal. In the valid camera list, the above-described capture error information is associated with each set of camera and capture PC. Therefore, a pair whose capture error information is 0 represents a pair whose state is normal. Further, when the capture error information is a value other than 0, it indicates that a failure has occurred in at least one of the camera and the capture PC of the set.

  In step S1008, the management server PC determines whether all capture PCs are normal. If the response from the capture PC 302 is impossible, or if there is an individual whose returned value is other than 0, the process proceeds to step S1009 to execute camera selection processing. The camera selection process is a process for generating screening control information for screening according to a scenario with a configuration excluding unusable cameras. Details of the camera selection process will be described later.

  In step S1010, the management server PC determines whether an error has occurred in the camera selection process. That is, it is determined whether or not the screening according to the scenario cannot be performed with the configuration excluding the unusable camera. If an error has occurred, the process proceeds to steps S1011 and S1012 to display an alert, and then instruct backup display to the image display PC 304 to end the process. As an example of the alert display here, it is possible to notify a capture PC in which a failure has occurred and to notify that it is switched to backup screening.

  In the present embodiment, the setting file of the application executed on all PCs constituting the screening system is placed in a shared folder disclosed on each PC. Therefore, the setting can be changed by accessing the shared folder of each PC from the management server. For example, the video buffering time in each PC, the backup screening instruction on the image display PC, and the like can be performed by changing such a shared folder setting file.

<Power controller>
FIG. 11 is a diagram illustrating a configuration example of the power supply controller 306. The operation of the power supply controller 306 will be described with reference to FIG. The power controller 306 acquires the valid camera list 1102 and the screening control information 1103 from the management server PC 301 and controls the power of the camera 101 and the capture PC 302 according to the information.

  FIG. 12 is a flowchart showing processing for restarting a device in which a failure has occurred in the power supply controller 306. In step S <b> 1201, the management unit 1101 first acquires the valid camera list 1102 from the management server PC 301. As described above, the valid camera list is a list indicating the states of the camera and the capture PC. In other words, the valid camera list can be said to be a list representing a set of a camera and a capture PC in which a failure has occurred.

  In step S1202, the management unit 1101 acquires the screening control information 1103 and stores the screening start time in the processing queue. In step S1203, the management unit 1101 waits until the screening start time with reference to the screening opening time stored in the processing queue. The reason for waiting until the screening start time is that the valid camera list and the screening control information may be updated before the screening starts. In this embodiment, in order to improve the stability of the system, the system is restarted after the screening start time with little influence of data update or the like has elapsed (at this time, the valid camera list and the screening control information are confirmed). That is, in this embodiment, the device in which the failure has occurred is not restarted unconditionally, but is restarted according to the state of the screening scenario. That is, after the start of the screening, a device that is not used in the screening scenario and has a failure is determined. Therefore, when it is determined that there is no influence on the screening scenario, the device in which the failure has occurred is automatically restarted individually. The reason why the restart is performed during the screening time is that the restart is completed before the next screening control information generation and is included in the valid camera list when the next screening control information is generated. It is not always necessary to restart after the start of the screening. If the effective camera list and screening control information used to generate the virtual viewpoint video to be screened is confirmed, the restart should be performed at any timing. Also good. In other words, the image data used for generating the virtual viewpoint video may be restarted after setting.

  When the screening starts, in step S1204, the management unit 1101 refers to the valid camera list 1102 and instructs the power supply processing unit 1103 to restart the device in which the failure has occurred. As described above, the valid camera list includes a value indicating a status in association with a pair of a camera and a capture PC. The status indicates the state of the camera and the capture PC. For example, when the status is 1, the capture PC cannot recognize the camera. Therefore, the power processing unit 1103 instructs the corresponding capture PC power management unit 1105 to restart. A status of 2 indicates that there are many capture errors. Therefore, the power processing unit 1103 instructs the corresponding camera monitoring unit 1104 to restart. When the status is 0, the capture PC and the camera are operating normally, so the restart is not performed. In this way, the power sources of the capture PC and the camera can be individually managed according to the status. The statuses that the system can take are not limited to this, and other statuses can also be taken. Depending on the status, the capture PC and the camera can be restarted at the same time.

  In step S1205, the power supply processing unit 1103 confirms that the restart of all the devices instructed in step S1204 is completed, and notifies the management unit 1101 of the completion of the restart.

  In the above example, the example in which the power supply controller 306 acquires the valid camera list and the screening control information and issues a restart instruction has been described. However, for example, the management server PC 301 may acquire the valid camera list and the screening control information, determine the target and timing for restarting, and notify the power controller 306 of it. Then, the power supply controller 306 may issue a restart instruction according to the notified contents.

<Virtual viewpoint video generation processing>
Next, the virtual viewpoint video generation processing in step S707 of this embodiment will be described. In the virtual viewpoint video generation process, a video with a constant height is used as a reference plane, and an image is generated so that objects on the reference plane are smoothly connected when the camera is switched. In this virtual viewpoint video generation process, the following video correction, conversion, and video effect addition are performed consistently.

(1) Camera image distortion and magnification chromatic aberration correction (2) Peripheral light amount correction (3) Perspective conversion between virtual viewpoint image and camera image (4) Addition of peripheral light amount drop effect to virtual viewpoint image (5) Addition of Distortion Effect This virtual viewpoint video generation process is efficient because it performs the above operations consistently, and is a process suitable for real-time performance. Note that if the processing time of the virtual viewpoint video generation processing is longer than the interval at which frames are reproduced on the image display PC 304 (for example, 1/30 seconds in the case of displaying 30 frames per second), real-time reproduction becomes impossible. In this embodiment, a plurality of image processing PCs 303 are prepared and interleaved. For example, when ten image processing PCs 303 are prepared, the virtual viewpoint video generation process may be performed in less than 1/3 second.

  FIG. 13 is a flowchart showing virtual viewpoint video generation processing performed in the virtual viewpoint video generation processing unit 605 of the image processing PC 303. Hereinafter, details of the virtual viewpoint video generation processing will be described using mathematical expressions.

  First, the physical position of the virtual camera

The direction of the virtual camera

A vector pointing up on the virtual camera screen

A vector pointing to the right direction of the virtual camera screen

And FIG. 14 is a diagram for explaining each parameter used in the virtual viewpoint video generation processing. Please also refer to FIG.

Is a vector that only represents the direction, so the length is 1. As the coordinate system representing these vectors, the above-described coordinate system having the floor surface as the xy plane and the height direction as the z axis is used. The horizontal half field angle and the vertical half field angle of the virtual camera are θ _{h, out} and θ _{v, out} , respectively.

The virtual viewpoint video generation process proceeds for each pixel of the virtual viewpoint video. here,
The coordinates of the pixel whose pixel value is to be determined

And First, in order to give a distortion effect to the virtual viewpoint video, in step S1301, reverse transformation (distortion correction) of the distortion effect is performed.

Pixel coordinates after distortion correction of virtual viewpoint video

Calculate A specific mathematical formula for the inverse transformation depends on what kind of distortion effect is given, but for example, the following formula (1) may be used.

here,

Is the center pixel position of the arbitrary viewpoint video.

Is a parameter that controls the strength of the distortion effect.

  Step S1302

Pixel

に写り、かつ前述の基準面上にある点の、３次元空間上の位置

The position in the three-dimensional space of the point on the reference plane described above

を求めるステップである。以下、基準面の高さをＺ_baseとする。

This is a step for obtaining. Hereinafter, the height of the reference plane is set as Z _base .

position

Is obtained by the following formulas (2) to (5).

Where X _{p, z} and X _{pixel, z} are respectively

and

Z component.

  In step S1303, in the c-th camera (hereinafter referred to as camera c),

position

Is reflected

Ideal pixel position

This is a step for obtaining. The ideal pixel position mentioned here is a pixel position when there is no aberration such as distortion or lateral chromatic aberration in the image of the camera c.

Ideal pixel position

は以下の式（６）〜（９）により求められる。

Is obtained by the following equations (6) to (9).

Here, X _{p, c, x} , X _{p, c, y} , X _{p, c, z} in equation (6) are respectively

X, y, and z components, and θ _{h, c} , θ _{v, c} are the horizontal and vertical half angles of view of the camera c. Of formula (8)

Is the position of the camera c. Further, the equation (9)

Are direction vectors representing the right direction, the upward direction, and the direction of the camera c, respectively. The length of these vectors is 1. These series of operations are operations in which three conversion operations generally called view conversion, projective conversion, and screen conversion are combined.

  In step S1304, taking into account the distortion and lateral chromatic aberration of the camera c,

Ideal pixel position

For each color,

Actual pixel position

Convert to Here, i represents a color index. This operation can be represented by the following formula (10) in terms of form.

The specific form of this conversion depends on the optical system of the camera c to be used. In general, this conversion cannot be expressed by a simple function. For this reason, this conversion is performed by referring to the table based on the actual measurement value.

  In step S1305, the camera c

Pixel position

The pixel values I _{p, c, i at} are obtained.

Pixel position

Since is a decimal, pixel values interpolated from surrounding pixels are obtained using bicubic interpolation or the like.

In step S1306, factors D _{p, c, i} for adding a peripheral light amount drop effect to the virtual viewpoint image are calculated while correcting the peripheral light amount drop of the camera c. This factor is expressed by the following formula (11): the amount of light drop effect C _p of the virtual start image,

Pixel position

Is defined as a ratio to the correction amount C _{p, c, i} of the peripheral light amount drop of the camera c.

The correction amount C _{p, c, i for} the peripheral light amount drop of the camera c cannot be generally expressed by a simple function, like the distortion correction. For this reason, a correction table is created based on the actually measured values, and correction is performed by referring to the table. This is formally expressed by the following formula (12).

As shown in the above equation, correction is performed in consideration of the dependency of the peripheral light amount drop on the pixel value.

  For example, the following expression (13) is used for the effect of the decrease in the amount of peripheral light added to the virtual viewpoint video. Here, β is a parameter for controlling the intensity of the peripheral light amount drop.

ステップＳ１３０７にて、仮想視点映像の画素値Ｉ_{ｏｕｔ，ｉ}を式（１４）にて計算する。

In step S1307, the pixel value I _{out, i} of the virtual viewpoint video is calculated by the equation (14).

Through the processing up to step S1307,

Pixel position of virtual viewpoint video

The pixel value I _{out, i} for color i is determined. Steps S1301 to S1307 are repeated until all the pixels of the virtual viewpoint video are processed in step S1308. Thereby, a virtual viewpoint video is generated.

  Here, in the virtual viewpoint video generation process, coordinates outside the captured image may be referred to. Such a region is dealt with by displaying a predetermined image. For example, in this embodiment, a floor texture image prepared in advance is used after being parse-converted according to a virtual viewpoint. By setting the pixel value when referring to the outside of the captured image as the pixel value of the floor texture image after the perspective conversion, it is possible to obtain an output image without a sense of incongruity.

<Camera selection process>
The camera selection process in the management server PC in step S1009 is executed based on the coverage ratio regarding the camera 101 that is operating normally. FIG. 15 is a diagram illustrating the coverage rate. The coverage is the ratio at which the image captured by the camera is included in the output image obtained by the virtual viewpoint video generation process. Accordingly, the cover ratio is calculated based on the height of the reference plane of the target frame. In FIG. 15A, all the pixels of the output image are obtained from the pixels of the captured image, and the coverage is 100%. On the other hand, in FIG. 15B, a partial area of the output image refers to outside the imaging range, and the coverage is less than 100%. In the virtual viewpoint video generation process, it is preferable to use a camera with a coverage rate of 100%. This is because, when the coverage is not 100%, an area that cannot be captured by the camera is included, leading to deterioration of the image quality of the virtual viewpoint video. Therefore, in this embodiment, a camera with a coverage rate of 100% is selected.

  FIG. 16 is a flowchart showing camera selection processing in the management server PC. In step S1601, the management server PC confirms whether the combination of the camera 101 and the capture PC 302 is operating normally. In this embodiment, an effective camera list is acquired. The valid camera list uses the list generated in step S1006.

  Next, in step S1602, the management server PC calculates the coverage of each camera registered in the valid camera list for each frame from the screening start time to the screening end time with reference to the screening control information. That is, the coverage rate of each camera is calculated using the screening control information and the valid camera list.

  Next, in step S1603, the management server PC executes a selection process for the camera with the 100% coverage calculated in step S1602. The selection process will be described later. It is determined whether the camera selected in step S1604 exists. If no camera has been selected by the selection process, the process advances to step S1605 to perform determination by reading the selection process mode setting. The selection processing mode setting is a setting as to whether or not a camera with a coverage rate of 100% is to be selected. When the selection processing mode is to be selected, a threshold Tx (%) is designated.

  If it is determined in step S1605 that a camera whose cover rate is not 100% is to be selected, the process proceeds to step S1606, and selection processing is executed for a camera whose cover rate is equal to or higher than Tx%. If the determination in step S1605 does not target a camera whose coverage is not 100%, and if no camera is selected in the selection process in step S1606 (No in S1607), an error is notified in step S1611 and the process ends. . The error in step S1611 is a notification indicating that the camera could not be selected.

  In step S1608, it is determined whether all frames have been selected. If all frames have been selected, the process proceeds to step S1609; otherwise, the process proceeds to step S1602.

  If the camera can be selected for all frames according to the screening control information, the management server PC executes a camera reselection process in step S1609. For example, when the transmission processing of video data for one frame does not end within 1/30 seconds, real-time screening cannot be performed if transmission processing from a specific capture PC continues for a long time. In addition, if the selected camera is frequently switched, the image quality between frames also changes and appears as flickering of the screened video. Therefore, it is necessary to make the request to the specific camera not continue for a long time while reducing the switching frequency of the camera as much as possible. Details of the reselection processing will be described later.

  In step S1610, the management server PC determines whether or not the reselection process has been completed normally. If an error has occurred in the reselection process, the process advances to step S1611 to notify the error and end the process. When an error is notified in the camera selection process, it is determined that an appropriate image quality cannot be obtained by normal screening, and the screen is automatically switched to backup screening as described in step S1011.

<Selection process>
Next, the selection processing in steps S1603 and S1606 will be described. In the selection process, first, the physical coordinates of the gazing point Pf in the target frame f and the vector Vf from the virtual camera to the gazing point Pf are calculated. The inner product maximum value Imax is initialized to zero. Next, in the selection process of step S1603, the process described below is executed for a camera with a coverage rate of 100% among the cameras registered in the valid camera list. On the other hand, in the selection process in step S1606, the process described below is executed for a camera having a coverage ratio of Tx% or more.

  First, a unit direction vector Vc from the camera c to the gazing point Pf is calculated. The inner product of the vector Vc and the vector Vf is taken, and the camera c having the maximum inner product is selected. By performing such processing, the camera closest to the viewpoint from the virtual camera can be selected.

<Reselection process>
Next, the reselection process in step S1609 will be described. In the screening system according to the present embodiment, cameras are arranged along the movement route of the virtual viewpoint. Therefore, the virtual viewpoint video is generated while sequentially switching and using the cameras.

  First, let c_sel be the camera selected at a certain time. Next, the camera c_prev selected immediately before is acquired. The camera selected immediately before refers not to the camera selected in the previous frame but to the camera selected immediately before the currently selected camera. Similarly, the camera c_prev2 selected two times before is acquired.

Subsequently, it is checked whether or not the same camera as the previous frame is selected. Processing differs depending on whether the same camera as the previous frame is selected.
(A) When the same camera as the previous frame is selected In this case, camera selection processing is performed in consideration of the number of continuous selections. In this embodiment, the upper limit of the number of continuous selections is N. First, if the number of continuous selections is less than N, this camera remains selected. If the request is continuously selected N times or more, it is confirmed whether the camera c_next to be selected next in the scenario can be substituted so that the request to the specific camera does not continue for a long time. Here, whether or not substitution is possible is determined based on the coverage. When the substitution is possible, the selected camera is changed to c_next for frames in which the camera c_sel is continuously selected after the currently processed frame. If the substitution is not possible, an error is notified and the process is terminated.
(B) When a camera different from the previous frame is selected In this case, camera selection processing is performed in consideration of the camera switching frequency. First, it is determined whether the selected camera c_sel is the same as the camera c_prev2 selected two times before. In the same case, there is a possibility that the selected camera is alternately switched between c_sel and c_prev, and therefore processing for avoiding this is executed.

  First, it is confirmed whether the camera c_prev is used continuously M times or more at the stage of the immediately preceding frame. If the camera has not been used more than M times consecutively, the camera switching frequency may be high, so it is confirmed whether c_prev can be used instead of c_sel. If c_prev can be substituted, the selected camera is changed to c_prev.

  Otherwise, it tries to substitute c_prev2 for the previous frame. When c_prev2 can be substituted, the selected cameras of all frames in which c_prev is continuously selected before the previous frame are changed to c_prev2. If it cannot be substituted, an error is notified and the process is terminated.

  These processes are repeated until all the cameras have been processed. When an error is notified in the reselection process, the screen is automatically switched to the backup screening.

  In the camera reselection process described in the present embodiment, as a result of applying the process (b), there is a possibility that the number of continuous selections is N or more. Therefore, it goes without saying that the number of continuous selections may be confirmed after this processing, and this state may be avoided by processing similar to (a).

  According to the present embodiment described above, when a failure occurs in a part of the camera or the capture PC, it becomes possible to restart the device during a scenario in which those devices are not used and use it at the next screening. . As a result, it is possible to construct a system capable of generating a virtual viewpoint video stably over a long period of time.

In the first embodiment, an example has been described in which a device in which a failure has occurred during scenario presentation is restarted. Here, when the influence of the device failure has little influence on the video presentation, the restart may be instructed only when a specific restart condition is satisfied. FIG. 17 is a flowchart showing an example of switching processing in the power supply controller 306 in accordance with the restart condition. Steps S1701 and S1702
S1703, S1705, and S1706 are the same as steps S1201 to S1205 in FIG. In this embodiment, in step S1704, it is determined whether or not an instruction to restart the device is given under a desired condition. The restart condition can be arbitrarily set, for example, when the number of effective cameras is 90% or less, or when the capture error is 10% or more. In addition, depending on the degree of failure, it may be possible that the failure cannot be resolved only by restarting. Therefore, in step S1704, whether or not the failure factor has been confirmed may be set as a restart condition. Since the processing other than the determination processing in step S1705 is the same as the flowchart described with reference to FIG.

<Other examples>
In the first and second embodiments, an example of a large-scale video reproduction system has been described. However, the present invention can also be applied to a device such as a multi-view camera. FIG. 18 is a schematic diagram showing the configuration of a multi-view camera. In FIG. 18, light received by the image sensor 1802 through the lens 1801 is converted into a digital signal via the A / D converter 1808. The converted data is recorded in the memory 1804 through the bus 1807. The recorded data is processed by the processor 1803 and displayed on the preview monitor 1805. In such a multi-lens camera configuration, the sensor power supply controller 1806 is provided, so that it is not necessary to initialize the entire camera.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  Further, the program code for realizing the function of the present embodiment may be executed by one computer (CPU, MPU), or may be executed by a plurality of computers cooperating. Also good. Further, the program code may be executed by a computer, or hardware such as a circuit for realizing the function of the program code may be provided. Alternatively, a part of the program code may be realized by hardware and the remaining part may be executed by a computer.

Claims (17)

A video playback system for playing back a virtual viewpoint video generated from image data captured by a plurality of imaging means,
Detecting means for detecting occurrence of a failure in the imaging means;
A control unit that restarts the imaging unit in which the occurrence of the failure is detected by the detection unit among the plurality of imaging units;
The control means starts the reproduction of the virtual viewpoint video generated from the image data picked up by the image pickup means other than the image pickup means other than the image pickup means in which the occurrence of the failure is detected. A video playback system that is restarted later.   The video reproduction system according to claim 1, wherein the imaging unit includes a camera and a device that transmits image data captured by the camera.   The video reproduction system according to claim 2, wherein the detection unit detects the occurrence of the failure based on responses from the camera and the device.   The detection means has the failure when there is no response from the camera, when there is no response from the device, or when there is a response that a capture error has occurred at a predetermined rate. The video reproduction system according to claim 3, wherein the video reproduction system is detected.   The video reproduction system according to any one of claims 2 to 4, wherein the detection unit determines a type of failure based on the detected content.   6. The video reproduction system according to claim 5, wherein the control unit restarts at least one power source of the camera or device in which the failure has occurred according to the type of the failure. A determination unit for determining a condition necessary for restarting the imaging unit;
The control unit is configured to set the imaging unit in which the failure has occurred after the start of reproduction of a virtual viewpoint video obtained from image data obtained by another imaging unit excluding the imaging unit, and to determine a condition in the determination unit. The video reproduction system according to any one of claims 1 to 6, wherein the video reproduction system is restarted when it is determined that it is satisfied. Imaging corresponding to image data used for generation of the virtual viewpoint video based on screening control information indicating imaging means corresponding to image data used for generation of each frame of the virtual viewpoint video and a detection result by the detection means The apparatus further comprises: selection means for selecting means; and playback means for playing back the virtual viewpoint video generated using the image data picked up by the image pickup means selected by the selection means. The video reproduction system according to any one of claims 7 to 9. When an effective imaging unit is not selected by the selection unit, the reproduction unit performs a backup screening,
9. The video reproduction system according to claim 8, wherein the control unit restarts the imaging unit in which the failure has occurred after the backup screening is started. The imaging means is a camera having a lens and an imaging sensor,
The video reproduction system according to claim 1, wherein the system is an imaging device including a multi-view camera. A plurality of image capturing means for capturing each image at a timing defined in a video playback scenario;
Detecting means for detecting occurrence of a failure in the imaging means;
A control unit that restarts the imaging unit in which the occurrence of the failure is detected by the detection unit, removing the timing of the imaging defined in the scenario;
A video playback system comprising: A video playback system for playing back a virtual viewpoint video generated from image data captured by a plurality of imaging means,
Detecting means for detecting occurrence of a failure in the imaging means;
Image data used to generate the virtual viewpoint video from among the image data captured by the imaging unit excluding the imaging unit in which the occurrence of the failure is detected, among the image data captured by the plurality of imaging units. A setting means for setting
And a control unit that restarts the imaging unit in which the occurrence of the failure is detected by the detection unit after setting of the image data used for generating the virtual viewpoint video by the setting unit. . A power management device in a video playback system for playing back a virtual viewpoint video generated from image data captured by a plurality of imaging means,
An acquisition unit that acquires information indicating an imaging unit in which a failure has occurred and information indicating a display start time of the virtual viewpoint video;
A power management apparatus, comprising: a control unit that restarts the power of the imaging unit in which the failure has occurred after the presentation start time of the virtual viewpoint video. A control method in a video playback system for playing back a virtual viewpoint video generated from image data captured by a plurality of imaging means,
A detection step of detecting the occurrence of a failure in the imaging means;
A control step of restarting the imaging unit in which the occurrence of the failure is detected by the detection unit among the plurality of imaging units,
The control step starts the reproduction of the virtual viewpoint video generated from the image data captured by the other imaging unit excluding the imaging unit in which the occurrence of the failure is detected. A control method which is a step of restarting later. A control method in a video playback system having a plurality of imaging means for capturing images at a timing specified in a video playback scenario,
A detection step of detecting the occurrence of a failure of the imaging means;
Re-starting the imaging means in which the occurrence of a failure is detected in the detection step, removing the timing defined in the scenario;
A control method characterized by comprising: A power management device in a video playback system for playing back a virtual viewpoint video generated from image data captured by a plurality of imaging means,
An acquisition step of acquiring information indicating an imaging unit in which a failure has occurred and information indicating a start time of the virtual viewpoint video;
And a step of restarting the power supply of the imaging means in which the failure has occurred after the elapse of the presentation start time of the virtual viewpoint video.   The program for functioning a computer as each means in the video reproduction system as described in any one of Claims 1-12.

Images