JP7467130B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Recently, a technology that has attracted attention is one in which multiple imaging devices are installed in different positions to synchronously capture images of a subject from multiple viewpoints, and a virtual viewpoint video is generated using a group of images from the multiple viewpoints (hereinafter, multiple viewpoint images) obtained by the capture. By using such a technology that generates a virtual viewpoint video from multiple viewpoint images, for example, it is possible to view highlight scenes of a soccer or basketball game from various angles, providing the user with a higher sense of realism than with normal images.

On the other hand, virtual viewpoint videos based on multiple viewpoint images are generated by collecting images captured by multiple cameras in an image processing unit such as a server, where the images are processed by the image processing unit to generate a three-dimensional model, render, and transmit the generated video to a user terminal. In a system that generates such virtual viewpoint videos, it is necessary to acquire multiple viewpoint images captured by multiple cameras with high accuracy and synchronized capture. Failure to ensure sufficient accuracy in synchronized capture will result in degradation of image quality in the generated virtual viewpoint video.

Patent document 1 describes a method for synchronizing the time of multiple cameras installed in different locations and connected in a daisy chain with high precision using a standard such as IEEE 1588, and capturing images synchronously from multiple viewpoints.

JP 2017-211828 A

However, with the method described in Patent Document 1, although highly accurate synchronized imaging is possible using a standard time synchronization method, if some of the imaging devices become out of sync, the image quality of the generated virtual viewpoint video deteriorates significantly. And even if a virtual viewpoint video of sufficient quality can be generated using only images captured by imaging devices other than the imaging device with the out-of-sync error, it was necessary to reset the entire system and resynchronize all imaging devices in order to maintain image quality.

In consideration of the above-mentioned problems, the present invention aims to prevent out-of-sync captured images from being used to generate a virtual viewpoint video in an imaging system that generates a virtual viewpoint video.

The information processing device of the present invention is an information processing device in an imaging system that generates a virtual viewpoint video using images based on imaging by multiple imaging devices, and is characterized in having an output means for outputting a first image based on imaging by a first imaging device among the multiple imaging devices to an external device, an acquisition means for acquiring first time information related to the first image and acquiring second time information related to a second image based on imaging by a second imaging device that is different from the first imaging device among the multiple imaging devices and captures images in synchronization with the first imaging device, and a control means for controlling the output means so as not to output the first image to the external device when the time indicated by the first time information and the time indicated by the second time information are different.

According to the present invention, in an imaging system that generates a virtual viewpoint video, it is possible to prevent an out-of-sync captured image from being used to generate the virtual viewpoint video.

FIG. 1 is a diagram illustrating an example of the configuration of an imaging system according to an embodiment. FIG. 2 is a diagram for explaining the positional relationship between a gaze point and a plurality of cameras. 2 is a block diagram showing an example of the hardware configuration of a camera adapter. 2 is a block diagram showing an example of a functional configuration of a camera adapter. 4 is a diagram showing an example of time information stored in a time information storage unit; FIG. FIG. 4 is a diagram showing an operation sequence of the imaging system according to the first embodiment. 11 is a flowchart illustrating an example of a processing procedure for determining whether or not the camera adapter transmits frame data in the first embodiment. 10A and 10B are diagrams for explaining the timing of transfer and transmission of frame data. FIG. 11 is a diagram showing an operation sequence of an imaging system according to a second embodiment. 13 is a flowchart illustrating an example of a processing procedure for determining whether or not the camera adapter transmits frame data in the second embodiment. FIG. 13 illustrates an example of an error packet. 10A and 10B are diagrams for explaining the timing of transfer and transmission of frame data. 11 is a diagram for explaining a procedure for determining whether or not a virtual viewpoint video can be generated by a server. FIG. 13 is a diagram for explaining the position of a camera that has become out of sync.

First Embodiment
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. In this embodiment, an example will be described in which a single gaze point is captured by a plurality of cameras, such as a case in which a plurality of cameras are installed in a facility such as a sports stadium or a concert hall to capture images.

Figure 1 is a diagram showing an example of the configuration of an imaging system 100 according to this embodiment. The imaging system 100 according to this embodiment has eight cameras 112a-h, corresponding camera adapters 120a-h, a switching hub 180, a server 200, a time server 290, and a controller 300. The cameras and camera adapters may be configured as one unit. The cameras may also have multiple imaging elements.

In the imaging system 100 of this embodiment, the controller 300, server 200, time server 290, camera adapter 120a, and camera adapter 120h are connected to the switching hub 180. The camera adapters 120a-h are connected to each other in a daisy chain. In this embodiment, the cameras 112a-h, which are imaging devices, are each connected to one camera adapter 120a-h, but the connection form is not limited to this. For example, a star-type network configuration may be used in which the camera adapters 120a-h are each connected to the switching hub 180, and data is sent and received between the camera adapters 120a-h via the switching hub 180.

The camera adapter 120a is connected to the camera 112a, and has functions such as controlling the camera 112a, acquiring captured images, providing synchronization signals, and setting the time. Controlling the camera 112a includes, for example, setting and referencing imaging parameters (such as the number of pixels, color depth, frame rate, and white balance settings), and acquiring the status of the camera 112a (capturing, stopped, synchronizing, error, etc.). Controlling the camera 112a also includes issuing instructions to start and stop imaging, and to adjust the focus, etc.

The camera adapter 120a synchronizes the imaging time by providing the imaging timing (control clock) to the camera 112a, which is the time synchronized with the time server 290. The time is set by the camera adapter 120a providing the time synchronized with the time server 290, for example, in a time code that conforms to the SMPTE12M format. This causes a time code to be added to the captured image received from the camera 112a. Note that the time code format is not limited to SMPTE12M, and other formats may be used. Also, it is assumed that the camera adapters 120b-h have the same functions as the camera adapter 120a.

The controller 300 controls the imaging of the cameras 112a-h by sending control signals to the camera adapters 120a-h via the network of the imaging system 100. In addition, when images captured by the cameras 112a-h are transferred to the server 200 via the network, the controller 300 selects and extracts the images from the server 200.

The server 200 receives and stores the captured images transmitted from the camera adaptors 120a-h. If no captured images are transmitted from any of the camera adaptors 120a-h, the server 200 determines whether or not it is possible to generate a virtual viewpoint image, and notifies the controller 300 of the result. The server 200 may be configured to form three-dimensional shape data of the subject contained in the image based on the transferred image, and further to generate a virtual viewpoint video. The virtual viewpoint video is generated by acquiring information indicating the position of the virtual viewpoint and the direction from the virtual viewpoint, and generating a virtual viewpoint video corresponding to the virtual viewpoint according to the information.

Figure 2 is a diagram for explaining the positional relationship between the gaze point 201 and the cameras 112a-h in the imaging system 100 shown in Figure 1. As shown in Figure 2, the cameras 112a-h and the camera adapters 120a-h are installed to surround the field 210, and the optical axis of each of the cameras 112a-h is directed toward the gaze point 201.

Next, the hardware configuration of the camera adapter 120a, which is an information processing device, will be described with reference to FIG. 3. The hardware configurations of the camera adapters 120b-h are the same as the configuration of the camera adapter 120a described below. The camera adapter 120a has a CPU 311, a ROM 312, a RAM 313, an auxiliary storage device 314, a communication I/F 317, and a bus 318.

The CPU 311 realizes the various functions of the camera adapter 120a shown in FIG. 4 described below by controlling the entire camera adapter 120a using computer programs and data stored in the ROM 312 and the RAM 313. The camera adapter 120a may have one or more dedicated hardware components different from the CPU 311, and at least a part of the processing by the CPU 311 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor). The ROM 312 stores programs that do not require modification. The RAM 313 temporarily stores programs and data supplied from the auxiliary storage device 314, and data supplied from the outside via the communication I/F 317. The auxiliary storage device 314 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

The communication I/F 317 is used for communication between the camera adapter 120a and an external device. For example, if the camera adapter 120a is connected to an external device via a wired connection, a communication cable is connected to the communication I/F 317. If the camera adapter 120a has a function for wireless communication with an external device, the communication I/F 317 is equipped with an antenna. The bus 318 connects each part of the camera adapter 120a to transmit information.

Figure 4 is a block diagram showing an example of the functional configuration of camera adapter 120a. Note that the functional configurations of camera adapters 120b-h are similar to the configuration of camera adapter 120a described below. Camera adapter 120a has a packet generation unit 421, a receiving unit 422, a transmitting unit 423, and a transmission determination unit 430.

The packet generation unit 421 receives the captured image captured by the camera 112a via the communication I/F 317 and generates a packet as payload data. The reception unit 422 receives the captured image transferred from the adjacent camera adapter in packet units via the communication I/F 317. The transmission unit 423 transmits the captured image in packet units to the adjacent camera adapter via the communication I/F 317.

The transmission determination unit 430 determines whether the captured image is to be transmitted in packets to the adjacent camera adapter by the transmission unit 423. The transmission determination unit 430 further includes a time information acquisition unit 431, a time information holding unit 432, and a time information determination unit 433.

The time information acquisition unit 431 acquires time information from a packet received by the receiving unit 422 from an adjacent camera adapter. The time information storage unit 432 stores the time information acquired by the time information acquisition unit 431 as part of the auxiliary storage device 314. The time information determination unit 433 determines whether there is an abnormality by comparing the time indicated by the time information of the packet generated by the packet generating unit 421 with the time indicated by the time information of the packet received by the receiving unit 422.

Figure 5 is a diagram showing an example of time information stored in the time information storage unit 432. In order to make the explanation easier to understand, the camera ID is the same as the symbol in Figure 5. The example shown in Figure 5 shows time information acquired by the time information storage unit 432 of the camera adapter 120e receiving the captured images of the cameras 112a-d. In this embodiment, as described later, only the time information of the captured images transferred from the camera adapters 120a-d located on the upstream side of the network is stored, and the captured images are not transferred from the downstream camera adapters 120f-h. Therefore, the time information storage unit 432 of the camera adapter 120e stores only the time information of the captured images of the cameras 112a-d. Also, as described later, the time information of the captured images received from the packet generation unit 421 is not stored in the time information storage unit 432.

Figure 6 is a diagram showing the operation sequence of the imaging system 100 in this embodiment. In this embodiment, an image captured by the camera 112a is transferred from the camera adapter 120a to the downstream camera adapters 120b-h via the daisy-chained network. The captured image is then stored in the server 200 via the switching hub 180. An image captured by the camera 112b is transferred from the camera adapter 120b to the downstream camera adapters 120c-h via the daisy-chained network. The captured image is then stored in the server 200 via the switching hub 180. Then, by a similar procedure, the captured images captured by each of the cameras 112c-h are stored in the server 200.

In addition, in this embodiment, the time information stored in the time information storage unit 432 is basically only the time information included in the packet received by the receiving unit 422. Note that in the example shown in FIG. 6, to simplify the explanation, a sequence for cameras 112a-c and camera adapters 120a-c is illustrated, but the same applies when cameras 112d-h and camera adapters 120d-h are also included.

First, in T601, the controller 300 sets camera parameters, which are image capturing conditions, in the cameras 112a to 112h via the camera adapters 120a to 120h.
Next, in T602, the controller 300 sets, to the camera adapters 120a to 120h, transmission parameters that determine the timing for starting transmission of images captured by the cameras 112a to 112h.

Next, in T603, the controller 300 performs time synchronization of the camera adapters 120a-h and the cameras 112a-h. For example, by performing time synchronization in compliance with IEEE 1588-2008, which is a standard for time synchronization, the camera adapters 120a-h and the cameras 112a-h can be synchronized with high accuracy.
Next, in T604, the controller 300 instructs the cameras 112a to 112h to start capturing images via the camera adapters 120a to 120h.

At T605, each of the cameras 112a to 112h captures an image for one frame period.
Then, in T606, the cameras 112a-h transmit one frame of captured image (hereinafter, frame data) to the corresponding camera adapters 120a-h, respectively. The camera adapters 120a-h then obtain the frame data in the packet generation units 421 and generate packets.

At T607, camera adapter 120a transmits the frame data acquired from camera 112a to server 200 via camera adapters 120b-h through the daisy-chained network. In this embodiment, camera adapter 120a is located at the top of the daisy-chained network, so it does not determine whether the time information is correct, but always transmits the frame data assuming that it is correct. Camera adapter 120a may also acquire time information from the frame data from camera 112a and store it in time information storage unit 432, and time information determination unit 433 may determine whether there is an anomaly in the time information based on its consistency with past time information.

On the other hand, at T608, the camera adapters 120b-h receive the frame data transferred from the camera adapter 120a at the receiving unit 422, and acquire time information from the frame data at the time information acquiring unit 431. The time information acquired by the time information acquiring unit 431 is then stored in the time information holding unit 432. Note that the frame data captured by the camera 112a received at the receiving unit 422 is not subject to the time information anomaly check, and is therefore always transferred as is by the transmitting unit 423 to the adjacent camera adapter, and is ultimately transmitted to the server 200.

Next, in T609, the camera adapter 120b acquires time information from the frame data from the camera 112b, and the transmission judgment unit 430 judges whether or not there is an anomaly in the time information of the frame data from the camera 112b. In this judgment, it is judged whether or not the time information acquired in T608 from the frame data from the camera 112a stored in the time information storage unit 432 matches the time information acquired in T609 via the packet generation unit 421 (whether or not the image capture times are the same). If the time information does not match (the image capture times are different), it is judged that there is an anomaly in the time information acquired in T609. If the time information matches, in T610, the transmission unit 423 of the camera adapter 120b transmits the frame data acquired from the camera 112b to the server 200 via the camera adapters 120c-h through the daisy-chained network. Note that if the time information does not match, the transmission unit 423 discards the frame data from the camera 112b without transmitting it.

At T611, camera adapters 120c-h receive the frame data transferred from camera adapter 120b at the receiving unit 422, and acquire time information from the frame data at the time information acquiring unit 431. The time information acquired by the time information acquiring unit 431 is then stored in the time information holding unit 432. Similarly, the frame data captured by camera 112b received at the receiving unit 422 is not subject to the time information anomaly check, and is therefore always transferred directly by the transmitting unit 423 to the adjacent camera adapter, and is ultimately sent to the server 200.

Next, at T612, the camera adapter 120c acquires time information from the frame data from the camera 112c, and the transmission judgment unit 430 judges whether or not there is an anomaly in the time information of the frame data from the camera 112c. In this judgment, it is judged whether or not the time information of the frame data from the cameras 112a and 112b acquired at T608 and T611 matches the time information of the frame data from the camera 112c acquired at T612. If the time information does not match, it is judged that there is an anomaly in the time information acquired at T612. If the time information matches, at T613, the transmission unit 423 of the camera adapter 120c transmits the frame data acquired from the camera 112c to the server 200 via the camera adapters 120d-h through the daisy-chained network. Note that if the time information does not match, the transmission unit 423 discards the frame data from the camera 112c without transmitting it.

Following the above process, the camera adapters 120d-h similarly obtain time information from the frame data sent by the cameras 112d-h, compare it with the time indicated by the time information stored in the time information storage unit 432, and if there is a match, transmit the frame data.

At T614, the cameras 112a-h each capture the next frame period in the same way as in T605. Then, at T615, the cameras 112a-h transmit one frame's worth of frame data to their corresponding camera adapters 120a-h. The camera adapters 120a-h then acquire the frame data in the packet generation unit 421 and generate packets. Thereafter, the same operations as in T607-T613 are performed. As described above, the same operations as in the first frame are repeated for the second and subsequent frames.

FIG. 7 is a flowchart showing an example of a processing procedure for determining whether or not the camera adapters 120b-h transmit frame data in this embodiment.
In step S701, the process waits until frame data is acquired in the packet generating unit 421. Then, when the frame data is acquired, the process proceeds to the next step S702.

In step S702, the time information determination unit 433 waits until it is time to transmit the frame data based on the transmission parameters set by the controller 300. Then, when it is time to transmit the frame data, the process proceeds to the next step S703.

In step S703, the time information determination unit 433 obtains time information from the frame data obtained by the packet generation unit 421, and determines whether or not the time information matches the time information of other cameras stored in the time information storage unit 432. If the result of this determination is that the time information matches, the process proceeds to step S704, and if the time information does not match, the process proceeds to step S705.

In step S704, the transmitting unit 423 transmits the frame data acquired in the packet generating unit 421 onto the daisy-chained network in packet units, and then the process returns to step S701.
On the other hand, in step S705, the time information determination unit 433 discards the frame data acquired in the packet generation unit 421, and the process returns to step S701.

Figure 8 is a diagram for explaining the timing of frame data transfer and transmission in this embodiment. Figure 8 shows the timing of synchronous imaging from frame N to frame N+6 and the transfer and transmission of frame data one frame later, and will be explained below using the processing in the camera adapter 120e as an example.

When synchronous imaging of the next frame (frame N+1) following synchronous imaging of frame N is performed, the receiving unit 422 of the camera adapter 120e receives the transferred frame data (frame N) of the cameras 112a-d from the daisy-chained network. At that time, the time information acquisition unit 431 acquires time information associated with the transferred frame data, and the acquired time information is stored in the time information storage unit 432. In other words, the frame data transmitted at T607, T610, etc. in FIG. 6 is received, and the time information is acquired at T608, T611, etc. Note that the frame data transfer period shown in FIG. 8 is for the camera adapter 120e, but the frame data transfer period is shorter in the case of the camera adapter 120b, and longer in the case of the camera adapter 120h.

Next, the transmitting unit 423 transmits frame data from camera 112e in response to a transmission start signal based on the transmission parameters set by the controller 300. At this time, the time information determining unit 433 determines whether the time information of the frame data from camera 112e matches the time information stored in the time information storing unit 432, and transmits only if they match. After that, the frame data from cameras 112f-h is transferred to the server 200, but the camera adapter 120e does not become a transfer path.

As described above, according to this embodiment, the frame data of a camera located upstream of the camera itself on a daisy-chained network is received to obtain time information. Then, only when the time information matches that of the frame data from the camera connected to the camera itself, the frame data is transmitted to the server via the downstream camera adapter. Even if some cameras experience image capture synchronization issues, the frame data is discarded and not transmitted if it differs from the time information of the upstream camera, so that it is possible to prevent frame data that has become out of sync from being transmitted to the server.

Here, if frame data is discarded, it may not be possible to generate a virtual viewpoint image. Therefore, we will explain the outline of the processing performed by the server 200 when a synchronization error occurs in the imaging of some of the cameras.

If frame data is not transmitted from some camera adapters, the server 200 determines whether it is possible to generate a virtual viewpoint video without using that frame data. This determination is made based on factors such as the position of the camera from which frame data was not transmitted.

When there are a sufficient number of cameras, the generation of the virtual viewpoint image is hardly affected even if frame data from one camera is missing. However, if adjacently placed cameras continuously lose synchronization, for example, as shown in FIG. 14, if a synchronization loss occurs in the imaging of cameras 112d and 112e, the image quality of the virtual viewpoint image will often deteriorate significantly or it will not be possible to generate it. Therefore, in the example shown in FIG. 14, it will be determined that the virtual viewpoint image cannot be generated. Furthermore, in addition to the position of the camera that has lost synchronization, the amount of deviation in position and time relative to the field or point of interest may also be taken into consideration.

When the server 200 determines that a virtual viewpoint image cannot be generated in this way, it notifies the controller 300 of that fact. When notified that a virtual viewpoint image cannot be generated, the controller 300 responds by resetting the entire imaging system 100, or by resetting the camera that has become out of sync and the camera adapter directly connected to that camera.

Second Embodiment
A second embodiment of the present invention will be described below. In this embodiment, similar to the first embodiment, frame data captured by cameras 112a-h is transferred to a downstream camera adapter via a daisy-chained network and stored in server 200 via switching hub 180. Also, in this embodiment, a packet including time information is transmitted from the camera adapter to all other camera adapters for each frame. Note that the configuration of the imaging system and the internal configuration of the camera adapters according to this embodiment are the same as those shown in Figs. 1-4, and therefore description thereof will be omitted. Below, differences from the first embodiment will be described.

Figure 9 is a diagram showing the operation sequence of the imaging system 100 in this embodiment. The sequence in this embodiment will be described below with reference to Figure 9. Note that in the example shown in Figure 9, in order to simplify the explanation, a sequence for cameras 112a-c and camera adapters 120a-c is illustrated, but the same applies when cameras 112d-h and camera adapters 120d-h are further included.

T901 to T906 are the same as T601 to T606 in Figure 6 described in the first embodiment.

At T907, the camera adapter 120a transmits a packet (hereafter referred to as the time information packet) containing time information obtained from the frame data from the camera 112a to the other camera adapters 120b-h. Here, since the camera adapter 120a is located at the most upstream position on the daisy-chained network, if it is transmitted in one direction, it will reach all the other camera adapters 120b-h in sequence, just like when transferring frame data.

On the other hand, camera adapter 120b is located second from the upstream on the daisy-chained network. Camera adapter 120b transmits time information packets obtained from frame data from camera 112b in two directions: to upstream camera adapter 120a and downstream camera adapters 120c-h. Similarly, camera adapter 120c transmits time information packets obtained from frame data from camera 112c in two directions: to upstream camera adapters 120a-b and downstream camera adapters 120d-h. Similarly, camera adapters 120d-h also transmit time information packets.

At T908, the camera adapters 120a-h receive the time information packet transmitted from the other camera adapters at the receiving unit 422, and acquire the time information from the packet at the time information acquisition unit 431. The time information acquired by the time information acquisition unit 431 is then held in the time information holding unit 432.
Next, at T909, the most upstream camera adapter 120a and the most downstream camera adapter 120h discard the time information packet.

Next, at T910, the camera adapters 120a-h acquire time information from the frame data acquired via the packet generation unit 421. Then, the transmission determination unit 430 judges whether or not there is an abnormality in the time information of the frame data acquired via the packet generation unit 421. In this judgment, it is judged whether or not the time information acquired at T908, which is held in the time information holding unit 432, matches the time information acquired at T910 via the packet generation unit 421. If the time information matches, at T911, the transmission unit 423 transmits the frame data acquired via the packet generation unit 421 to the server 200 via the downstream camera adapter. Note that if the time information does not match, the transmission unit 423 discards the frame data without transmitting it, and instead transmits an error packet to the server 200. As described above, the same operation as for the first frame is repeated for the second and subsequent frames.

FIG. 10 is a flowchart showing an example of a processing procedure for determining whether or not the camera adapters 120a to 120h transmit frame data in this embodiment.
In step S1001, the process waits until frame data is acquired in the packet generating unit 421. Then, when the frame data is acquired, the process proceeds to the next step S1002.

In step S1002, the transmitting unit 423 transmits a time information packet to the other camera adapter. Then, in step S1003, the receiving unit 422 receives the time information packet from the other camera adapter, and the time information acquiring unit 431 acquires the time information from the packet. At this time, the acquired time information is stored in the time information storing unit 432.

Next, in step S1004, the time information determination unit 433 obtains time information from the frame data obtained by the packet generation unit 421, and determines whether or not it matches the time information of other cameras stored in the time information storage unit 432. If the result of this determination is that the time information matches, the process proceeds to step S1005, and if the time information does not match, the process proceeds to step S1006.

In step S1005, the transmitting unit 423 transmits the frame data acquired in the packet generating unit 421 onto the daisy-chained network in packet units, and then returns to step S1001.
On the other hand, in step S1006, the time information determination unit 433 discards the frame data acquired by the packet generation unit 421 and generates an error packet. Then, in step S1007, the transmission unit 423 transmits the error packet to the server 200, and the process returns to step S1001. Note that Fig. 11 shows an example of an error packet generated by the time information determination unit 433 of the camera adapter 120a, and the error packet includes a camera ID and error information.

Figure 12 is a diagram for explaining the transfer and transmission of frame data and the timing of the transfer in this embodiment. In the first embodiment, frame data was transferred and transmitted sequentially from one of the camera adapters 120a-h to another, but in this embodiment, frame data is transferred and transmitted at approximately the same timing for all of the camera adapters 120a-h. Note that frame data may also be transferred and transmitted sequentially from one camera adapter to another, as in the first embodiment.

In addition, in this embodiment, when the server 200 receives an error packet, it determines whether or not a virtual viewpoint image can be generated, similar to the first embodiment. Figure 13 is a diagram for explaining the procedure by which the server 200 determines whether or not a virtual viewpoint image can be generated in this embodiment. As shown in Figure 13, when the server 200 receives frame data from the camera adapters 120a-h and then receives an error packet from some of the camera adapters, the server 200 determines whether or not a virtual viewpoint image can be generated. The criteria for this determination are the same as in the first embodiment.

If the server 200 determines that it is not possible to generate a virtual viewpoint video, it notifies the controller 300 accordingly. When notified that it is not possible to generate a virtual viewpoint video, the controller 300 responds by resetting the entire imaging system 100, or by resetting the camera adapter that sent the error packet and the corresponding camera.

Also, as shown in FIG. 13, if an error packet is not received, or if an error packet is received but it is determined that a virtual viewpoint video can be generated, the controller 300 may be notified that a virtual viewpoint video can be generated. This allows the controller 300 to confirm that there is no problem with generating a virtual viewpoint video as the imaging system 100.

As described above, according to this embodiment, the time information packets are received from other camera adapters on a daisy-chained network, and the time information of the other cameras is stored. Then, frame data is sent to the server only when the time information matches that of the frame data from the camera connected to the adapter itself. Even if some cameras become out of sync, if the time information differs from that of the other cameras, the frame data is discarded and an error packet is sent instead, so that out-of-sync frame data can be prevented from being sent to the server.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

421 packet generation unit, 422 receiving unit, 423 transmitting unit, 431 time information acquisition unit, 433 time information determination unit

Claims (11)

An information processing device in an imaging system that generates a virtual viewpoint video using images based on imaging by a plurality of imaging devices,
an output unit that outputs a first image based on an image captured by a first imaging device among the plurality of imaging devices to an external device;
an acquisition means for acquiring first time information related to the first image , and acquiring second time information related to a second image based on imaging by a second imaging device that is different from the first imaging device among the plurality of imaging devices and captures images in synchronization with the first imaging device;
An information processing device characterized by having a control means for controlling the output means so as not to output the first image to the external device when the time indicated by the first time information and the time indicated by the second time information are different. 2 . The information processing apparatus according to claim 1 , wherein the acquisition means acquires the second image output from another information processing apparatus, and acquires the second time information from the acquired second image . 3. The information processing apparatus according to claim 2, wherein the output means transfers the second image to the external device. 2 . The information processing apparatus according to claim 1 , wherein the acquiring means acquires the second time information related to the second image from a packet. the acquiring means acquires a plurality of pieces of time information related to a plurality of images output from a plurality of other information processing devices;
The information processing device according to claim 1, characterized in that the control means controls the output means so as not to output the first image to the external device based on a comparison between the time indicated by the multiple time information relating to the multiple images and the time indicated by the first time information. 6. The information processing apparatus according to claim 5 , wherein, when the comparison result shows that the time indicated by the first time information is different from the times indicated by the plurality of time information, the output means outputs error information to the external device. 7. The information processing apparatus according to claim 1, wherein the output unit outputs the first image to the external device connected in a daisy chain. The information processing device according to claim 1, characterized in that the external device is another information processing device. The information processing device according to claim 1, characterized in that the external device is a device that generates three-dimensional shape data of a subject. An information processing method of an information processing device in an imaging system that generates a virtual viewpoint video using images based on imaging by a plurality of imaging devices, comprising:
an output step of outputting a first image based on imaging by a first imaging device among the plurality of imaging devices to an external device;
an acquisition step of acquiring first time information related to the first image , and acquiring second time information related to a second image based on imaging by a second imaging device that is different from the first imaging device among the plurality of imaging devices and captures images in synchronization with the first imaging device;
and a control step of controlling the output step so as not to output the first image to the external device when the time indicated by the first time information is different from the time indicated by the second time information. A program for causing a computer to function as each of the means of an information processing device according to any one of claims 1 to 9.

Images